Re: Kylin with S3, cubes tables get in transition when new cluster booted

Moisés Català Mon, 30 Jul 2018 08:27:02 -0700

Thanks for the tips Roberto,

You’re right, when I deploy emr and install Kylie everything works like a charm, even I can build the sample cube.

I have added the config you suggested about using EMRFS in emrfs-site and I have launched a brand new cluster.

I also deployed Kylie and built the cube. Finally I shut down Kylin & disabled all hbase tables.

Unfortunately, when I launch a new cluster, hbase master node can’t boot, looking the log appears this:

2018-07-30 15:00:31,103 ERROR [ip-172-31-85-0:16000.activeMasterManager] consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item /da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:02:49,220 ERROR [ip-172-31-85-0:16000.activeMasterManager] consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item /da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:09:01,324 ERROR [ip-172-31-85-0:16000.activeMasterManager] consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item /da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:09:01,325 FATAL [ip-172-31-85-0:16000.activeMasterManager] master.HMaster: Failed to become active master
com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: 1 items inconsistent (no s3 object for associated metadata item). First object: /da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
	at com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:749)
	at com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:519)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy30.listStatus(Unknown Source)
	at com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.listStatus(S3NativeFileSystem2.java:206)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1532)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1558)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1603)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1597)
	at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFileSystem.java:347)
	at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1737)
	at org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:377)
	at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:358)
	at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:339)
	at org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.needsMigration(FSTableDescriptorMigrationToSubdir.java:59)
	at org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.migrateFSTableDescriptorsIfNecessary(FSTableDescriptorMigrationToSubdir.java:45)
	at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:526)
	at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:166)
	at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:725)
	at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:198)
	at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1907)
	at java.lang.Thread.run(Thread.java:748)
2018-07-30 15:09:01,326 FATAL [ip-172-31-85-0:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.

I have attached the full log to the email.

What am I missing???

Thanks in advance

2018-07-30 15:00:31,103 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:02:49,220 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:09:01,324 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:09:01,325 FATAL [ip-172-31-85-0:16000.activeMasterManager] 
master.HMaster: Failed to become active master
com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: 1 items 
inconsistent (no s3 object for associated metadata item). First object: 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
        at 
com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:749)
        at 
com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:519)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy30.listStatus(Unknown Source)
        at 
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.listStatus(S3NativeFileSystem2.java:206)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1532)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1558)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1603)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1597)
        at 
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFileSystem.java:347)
        at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1737)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:377)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:358)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:339)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.needsMigration(FSTableDescriptorMigrationToSubdir.java:59)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.migrateFSTableDescriptorsIfNecessary(FSTableDescriptorMigrationToSubdir.java:45)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:526)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:166)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
        at 
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:725)
        at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:198)
        at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1907)
        at java.lang.Thread.run(Thread.java:748)
2018-07-30 15:09:01,326 FATAL [ip-172-31-85-0:16000.activeMasterManager] 
master.HMaster: Unhandled exception. Starting shutdown.
com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: 1 items 
inconsistent (no s3 object for associated metadata item). First object: 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
        at 
com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:749)
        at 
com.amazon.ws.emr.hadoop.fs.consistency.ConsistencyCheckerS3FileSystem.listStatus(ConsistencyCheckerS3FileSystem.java:519)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy30.listStatus(Unknown Source)
        at 
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.listStatus(S3NativeFileSystem2.java:206)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1532)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1558)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1603)
        at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1597)
        at 
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFileSystem.java:347)
        at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1737)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:377)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:358)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:339)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.needsMigration(FSTableDescriptorMigrationToSubdir.java:59)
        at 
org.apache.hadoop.hbase.util.FSTableDescriptorMigrationToSubdir.migrateFSTableDescriptorsIfNecessary(FSTableDescriptorMigrationToSubdir.java:45)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:526)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:166)
        at 
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
        at 
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:725)
        at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:198)
        at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1907)
        at java.lang.Thread.run(Thread.java:748)
2018-07-30 15:09:01,327 INFO  [ip-172-31-85-0:16000.activeMasterManager] 
regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2018-07-30 15:09:03,072 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] ipc.RpcServer: Stopping 
server on 16000
2018-07-30 15:09:03,073 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: 
RpcServer.listener,port=16000: stopping
2018-07-30 15:09:03,073 INFO  [RpcServer.responder] ipc.RpcServer: 
RpcServer.responder: stopped
2018-07-30 15:09:03,073 INFO  [RpcServer.responder] ipc.RpcServer: 
RpcServer.responder: stopping
2018-07-30 15:09:03,073 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
regionserver.HRegionServer: Stopping infoServer
2018-07-30 15:09:03,084 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] mortbay.log: Stopped 
[email protected]:16010
2018-07-30 15:09:03,186 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
regionserver.HRegionServer: stopping server 
ip-172-31-85-0.ec2.internal,16000,1532962693799
2018-07-30 15:09:03,186 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
client.ConnectionManager$HConnectionImplementation: Closing zookeeper 
sessionid=0x164eb7429950018
2018-07-30 15:09:03,187 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] zookeeper.ZooKeeper: 
Session: 0x164eb7429950018 closed
2018-07-30 15:09:03,188 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
regionserver.HRegionServer: stopping server 
ip-172-31-85-0.ec2.internal,16000,1532962693799; all regions closed.
2018-07-30 15:09:03,188 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] hbase.ChoreService: 
Chore service for: ip-172-31-85-0.ec2.internal,16000,1532962693799 had 
[[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: 
MILLISECONDS]] on shutdown
2018-07-30 15:09:03,189 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000-EventThread] 
zookeeper.ClientCnxn: EventThread shut down for session: 0x164eb7429950018
2018-07-30 15:09:03,194 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] ipc.RpcServer: Stopping 
server on 16000
2018-07-30 15:09:03,196 INFO  [main-EventThread] zookeeper.ClientCnxn: 
EventThread shut down for session: 0x164eb7429950017
2018-07-30 15:09:03,196 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] zookeeper.ZooKeeper: 
Session: 0x164eb7429950017 closed
2018-07-30 15:09:03,196 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
regionserver.HRegionServer: stopping server 
ip-172-31-85-0.ec2.internal,16000,1532962693799; zookeeper connection closed.
2018-07-30 15:09:03,197 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
regionserver.HRegionServer: 
master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000 exiting
Mon Jul 30 15:09:03 UTC 2018 Starting master on ip-172-31-85-0
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 122653
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 122653
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2018-07-30 15:09:04,613 INFO  [main] util.VersionInfo: HBase 1.3.1
2018-07-30 15:09:04,613 INFO  [main] util.VersionInfo: Source code repository 
file:///workspace/workspace/bigtop.release-rpm-5.10.0/build/hbase/rpm/BUILD/hbase-1.3.1
 revision=Unknown
2018-07-30 15:09:04,613 INFO  [main] util.VersionInfo: Compiled by ec2-user on 
Thu Nov 16 09:03:35 UTC 2017
2018-07-30 15:09:04,613 INFO  [main] util.VersionInfo: From source with 
checksum f092524f1d30f04b9734e4362a96138b
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:BASH_FUNC_run_prestart()=() {  export HBASE_HOME=/usr/lib/hbase;
 BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default};
 [ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && . 
${BIGTOP_DEFAULTS_DIR}/hbase;
 su -s /bin/bash hbase -c "${EXEC_PATH} start master"
}
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HADOOP_CONF_DIR=/etc/hadoop/conf
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HBASE_PID_DIR=/var/run/hbase
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:PIDFILE=/var/run/hbase/hbase-hbase-master.pid
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HBASE_REGIONSERVER_OPTS=-Xmx11712m
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HBASE_CONF_DIR=/etc/hbase/conf
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: env:DESC=HBase 
master daemon
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:EXEC_PATH=/usr/lib/hbase/bin/hbase-daemon.sh
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: env:SLEEP_TIME=10
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HBASE_DAEMON_DEFAULT_SECURITY_LOGGER=INFO,DRFAS
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:LD_LIBRARY_PATH=::/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2018-07-30 15:09:05,002 INFO  [main] util.ServerCommandLine: 
env:HADOOP_HOME_WARN_SUPPRESS=true
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: env:LOGNAME=hbase
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_REST_OPTS=
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:JSVC_HOME=/usr/lib/bigtop-utils
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: env:PWD=/
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HADOOP_PREFIX=/usr/lib/hadoop
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_ROOT_LOGGER=INFO,DRFA
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: env:SHELL=/bin/bash
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:WORKING_DIR=/var/lib/hbase
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_HEAPSIZE=1024
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_ENV_INIT=true
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HADOOP_YARN_HOME=/usr/lib/hadoop-yarn
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_DAEMON_DEFAULT_ROOT_LOGGER=INFO,DRFA
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:HBASE_MANAGES_ZK=false
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: 
env:UPSTART_INSTANCE=
2018-07-30 15:09:05,003 INFO  [main] util.ServerCommandLine: env:DAEMON_FLAGS=
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HADOOP_HOME=/usr/lib/hadoop
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_NICENESS=0
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: env:HBASE_OPTS= 
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 
-Dsun.net.inetaddr.ttl=60 -Dnetworkaddress.cache.ttl=60   
-Dhbase.log.dir=/var/log/hbase 
-Dhbase.log.file=hbase-hbase-master-ip-172-31-85-0.log 
-Dhbase.home.dir=/usr/lib/hbase -Dhbase.id.str=hbase 
-Dhbase.root.logger=INFO,DRFA 
-Djava.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native 
-Dhbase.security.logger=INFO,DRFAS
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_START_FILE=/var/run/hbase/hbase-hbase-master.autorestart
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:DAEMON=hbase-master
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_SECURITY_LOGGER=INFO,DRFAS
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: env:SHLVL=4
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_LOGFILE=hbase-hbase-master-ip-172-31-85-0.log
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:UPSTART_JOB=hbase-master
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:JAVA_HOME=/usr/lib/jvm/java-openjdk
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:CONF_DIR=/etc/hbase/conf
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: env:TERM=linux
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:LANG=en_US.UTF-8
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HADOOP_COMMON_HOME=/usr/lib/hadoop
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_CLASSPATH=/etc/hadoop/conf
2018-07-30 15:09:05,004 INFO  [main] util.ServerCommandLine: 
env:HBASE_IDENT_STRING=hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HBASE_ZNODE_FILE=/var/run/hbase/hbase-hbase-master.znode
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HBASE_LOG_PREFIX=hbase-hbase-master-ip-172-31-85-0
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HBASE_LOG_DIR=/var/log/hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: env:USER=hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:CLASSPATH=/etc/hbase/conf:/usr/lib/jvm/java-openjdk/lib/tools.jar:/usr/lib/hbase:/usr/lib/hbase/lib/activation-1.1.jar:/usr/lib/hbase/lib/aopalliance-1.0.jar:/usr/lib/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hbase/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hbase/lib/api-util-1.0.0-M20.jar:/usr/lib/hbase/lib/asm-3.1.jar:/usr/lib/hbase/lib/avro-1.7.4.jar:/usr/lib/hbase/lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/lib/commons-cli-1.2.jar:/usr/lib/hbase/lib/commons-codec-1.9.jar:/usr/lib/hbase/lib/commons-collections-3.2.2.jar:/usr/lib/hbase/lib/commons-compress-1.4.1.jar:/usr/lib/hbase/lib/commons-configuration-1.6.jar:/usr/lib/hbase/lib/commons-daemon-1.0.13.jar:/usr/lib/hbase/lib/commons-digester-1.8.jar:/usr/lib/hbase/lib/commons-el-1.0.jar:/usr/lib/hbase/lib/commons-httpclient-3.1.jar:/usr/lib/hbase/lib/commons-io-2.4.jar:/usr/lib/hbase/lib/commons-lang-2.6.jar:/usr/lib/hbase/lib/commons-logging-1.2.jar:/usr/lib/hbase/lib/commons-math-2.2.jar:/usr/lib/hbase/lib/commons-math3-3.1.1.jar:/usr/lib/hbase/lib/commons-net-3.1.jar:/usr/lib/hbase/lib/curator-client-2.7.1.jar:/usr/lib/hbase/lib/curator-framework-2.7.1.jar:/usr/lib/hbase/lib/curator-recipes-2.7.1.jar:/usr/lib/hbase/lib/disruptor-3.3.0.jar:/usr/lib/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/lib/hbase/lib/gson-2.2.4.jar:/usr/lib/hbase/lib/guava-12.0.1.jar:/usr/lib/hbase/lib/guice-3.0.jar:/usr/lib/hbase/lib/guice-servlet-3.0.jar:/usr/lib/hbase/lib/hadoop-annotations.jar:/usr/lib/hbase/lib/hadoop-auth.jar:/usr/lib/hbase/lib/hadoop-common.jar:/usr/lib/hbase/lib/hadoop-hdfs.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-app.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-common.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-core.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hbase/lib/hadoop-yarn-api.jar:/usr/lib/hbase/lib/hadoop-yarn-client.jar:/usr/lib/hbase/lib/hadoop-yarn-common.jar:/usr/lib/hbase/lib/hadoop-yarn-server-common.jar:/usr/lib/hbase/lib/hbase-annotations-1.3.1.jar:/usr/lib/hbase/lib/hbase-annotations-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-client-1.3.1.jar:/usr/lib/hbase/lib/hbase-common-1.3.1.jar:/usr/lib/hbase/lib/hbase-common-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-examples-1.3.1.jar:/usr/lib/hbase/lib/hbase-external-blockcache-1.3.1.jar:/usr/lib/hbase/lib/hbase-hadoop2-compat-1.3.1.jar:/usr/lib/hbase/lib/hbase-hadoop-compat-1.3.1.jar:/usr/lib/hbase/lib/hbase-it-1.3.1.jar:/usr/lib/hbase/lib/hbase-it-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-prefix-tree-1.3.1.jar:/usr/lib/hbase/lib/hbase-procedure-1.3.1.jar:/usr/lib/hbase/lib/hbase-protocol-1.3.1.jar:/usr/lib/hbase/lib/hbase-resource-bundle-1.3.1.jar:/usr/lib/hbase/lib/hbase-rest-1.3.1.jar:/usr/lib/hbase/lib/hbase-server-1.3.1.jar:/usr/lib/hbase/lib/hbase-server-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-shell-1.3.1.jar:/usr/lib/hbase/lib/hbase-thrift-1.3.1.jar:/usr/lib/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hbase/lib/httpclient-4.5.3.jar:/usr/lib/hbase/lib/httpcore-4.3.2.jar:/usr/lib/hbase/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hbase/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hbase/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hbase/lib/jackson-xc-1.9.13.jar:/usr/lib/hbase/lib/jamon-runtime-2.4.1.jar:/usr/lib/hbase/lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/lib/javax.inject-1.jar:/usr/lib/hbase/lib/java-xmlbuilder-0.4.jar:/usr/lib/hbase/lib/jaxb-api-2.2.2.jar:/usr/lib/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hbase/lib/jcodings-1.0.8.jar:/usr/lib/hbase/lib/jersey-client-1.9.jar:/usr/lib/hbase/lib/jersey-core-1.9.jar:/usr/lib/hbase/lib/jersey-guice-1.9.jar:/usr/lib/hbase/lib/jersey-json-1.9.jar:/usr/lib/hbase/lib/jersey-server-1.9.jar:/usr/lib/hbase/lib/jets3t-0.9.0.jar:/usr/lib/hbase/lib/jettison-1.3.3.jar:/usr/lib/hbase/lib/jetty-6.1.26.jar:/usr/lib/hbase/lib/jetty-sslengine-6.1.26.jar:/usr/lib/hbase/lib/jetty-util-6.1.26.jar:/usr/lib/hbase/lib/joni-2.1.2.jar:/usr/lib/hbase/lib/jruby-complete-1.6.8.jar:/usr/lib/hbase/lib/jsch-0.1.42.jar:/usr/lib/hbase/lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/lib/junit-4.12.jar:/usr/lib/hbase/lib/leveldbjni-all-1.8.jar:/usr/lib/hbase/lib/libthrift-0.9.3.jar:/usr/lib/hbase/lib/log4j-1.2.17.jar:/usr/lib/hbase/lib/metrics-core-2.2.0.jar:/usr/lib/hbase/lib/netty-all-4.0.23.Final.jar:/usr/lib/hbase/lib/paranamer-2.3.jar:/usr/lib/hbase/lib/protobuf-java-2.5.0.jar:/usr/lib/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/lib/servlet-api-2.5.jar:/usr/lib/hbase/lib/slf4j-api-1.6.1.jar:/usr/lib/hbase/lib/snappy-java-1.0.4.1.jar:/usr/lib/hbase/lib/spymemcached-2.11.6.jar:/usr/lib/hbase/lib/xmlenc-0.52.jar:/usr/lib/hbase/lib/xz-1.0.jar:/usr/lib/hbase/lib/zookeeper.jar:/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/etc/tez/conf:/usr/lib/tez/*:/usr/lib/tez/lib/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/etc/hadoop/conf
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: env:SVC_USER=hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HBASE_THRIFT_OPTS=
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HBASE_HOME=/usr/lib/hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:HOME=/var/lib/hbase
2018-07-30 15:09:05,005 INFO  [main] util.ServerCommandLine: 
env:MALLOC_ARENA_MAX=4
2018-07-30 15:09:05,007 INFO  [main] util.ServerCommandLine: vmName=OpenJDK 
64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.171-b10
2018-07-30 15:09:05,007 INFO  [main] util.ServerCommandLine: 
vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx1024m, 
-XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, 
-XX:CMSInitiatingOccupancyFraction=70, -Dsun.net.inetaddr.ttl=60, 
-Dnetworkaddress.cache.ttl=60, -Dhbase.log.dir=/var/log/hbase, 
-Dhbase.log.file=hbase-hbase-master-ip-172-31-85-0.log, 
-Dhbase.home.dir=/usr/lib/hbase, -Dhbase.id.str=hbase, 
-Dhbase.root.logger=INFO,DRFA, 
-Djava.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native, 
-Dhbase.security.logger=INFO,DRFAS]
2018-07-30 15:09:05,387 INFO  [main] regionserver.RSRpcServices: 
master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000 server-side HConnection 
retries=350
2018-07-30 15:09:05,538 INFO  [main] ipc.SimpleRpcScheduler: Using fifo as user 
call queue, count=3
2018-07-30 15:09:05,555 INFO  [main] ipc.RpcServer: 
master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000: started 10 reader(s) 
listening on port=16000
2018-07-30 15:09:09,431 INFO  [main] fs.HFileSystem: Added intercepting call to 
namenode#getBlockLocations so can do block reordering using class 
org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2018-07-30 15:09:09,979 INFO  [main] zookeeper.RecoverableZooKeeper: Process 
identifier=master:16000 connecting to ZooKeeper 
ensemble=ip-172-31-85-0.ec2.internal:2181
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:zookeeper.version=3.4.10-d73c901b4228f4e75d3a527ec2318ce7376036cb, 
built on 11/16/2017 08:48 GMT
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:host.name=ip-172-31-85-0.ec2.internal
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.version=1.8.0_171
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.vendor=Oracle Corporation
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.class.path=/etc/hbase/conf:/usr/lib/jvm/java-openjdk/lib/tools.jar:/usr/lib/hbase:/usr/lib/hbase/lib/activation-1.1.jar:/usr/lib/hbase/lib/aopalliance-1.0.jar:/usr/lib/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hbase/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hbase/lib/api-util-1.0.0-M20.jar:/usr/lib/hbase/lib/asm-3.1.jar:/usr/lib/hbase/lib/avro-1.7.4.jar:/usr/lib/hbase/lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/lib/commons-cli-1.2.jar:/usr/lib/hbase/lib/commons-codec-1.9.jar:/usr/lib/hbase/lib/commons-collections-3.2.2.jar:/usr/lib/hbase/lib/commons-compress-1.4.1.jar:/usr/lib/hbase/lib/commons-configuration-1.6.jar:/usr/lib/hbase/lib/commons-daemon-1.0.13.jar:/usr/lib/hbase/lib/commons-digester-1.8.jar:/usr/lib/hbase/lib/commons-el-1.0.jar:/usr/lib/hbase/lib/commons-httpclient-3.1.jar:/usr/lib/hbase/lib/commons-io-2.4.jar:/usr/lib/hbase/lib/commons-lang-2.6.jar:/usr/lib/hbase/lib/commons-logging-1.2.jar:/usr/lib/hbase/lib/commons-math-2.2.jar:/usr/lib/hbase/lib/commons-math3-3.1.1.jar:/usr/lib/hbase/lib/commons-net-3.1.jar:/usr/lib/hbase/lib/curator-client-2.7.1.jar:/usr/lib/hbase/lib/curator-framework-2.7.1.jar:/usr/lib/hbase/lib/curator-recipes-2.7.1.jar:/usr/lib/hbase/lib/disruptor-3.3.0.jar:/usr/lib/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/lib/hbase/lib/gson-2.2.4.jar:/usr/lib/hbase/lib/guava-12.0.1.jar:/usr/lib/hbase/lib/guice-3.0.jar:/usr/lib/hbase/lib/guice-servlet-3.0.jar:/usr/lib/hbase/lib/hadoop-annotations.jar:/usr/lib/hbase/lib/hadoop-auth.jar:/usr/lib/hbase/lib/hadoop-common.jar:/usr/lib/hbase/lib/hadoop-hdfs.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-app.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-common.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-core.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-jobclient.jar:/usr/lib/hbase/lib/hadoop-mapreduce-client-shuffle.jar:/usr/lib/hbase/lib/hadoop-yarn-api.jar:/usr/lib/hbase/lib/hadoop-yarn-client.jar:/usr/lib/hbase/lib/hadoop-yarn-common.jar:/usr/lib/hbase/lib/hadoop-yarn-server-common.jar:/usr/lib/hbase/lib/hbase-annotations-1.3.1.jar:/usr/lib/hbase/lib/hbase-annotations-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-client-1.3.1.jar:/usr/lib/hbase/lib/hbase-common-1.3.1.jar:/usr/lib/hbase/lib/hbase-common-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-examples-1.3.1.jar:/usr/lib/hbase/lib/hbase-external-blockcache-1.3.1.jar:/usr/lib/hbase/lib/hbase-hadoop2-compat-1.3.1.jar:/usr/lib/hbase/lib/hbase-hadoop-compat-1.3.1.jar:/usr/lib/hbase/lib/hbase-it-1.3.1.jar:/usr/lib/hbase/lib/hbase-it-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-prefix-tree-1.3.1.jar:/usr/lib/hbase/lib/hbase-procedure-1.3.1.jar:/usr/lib/hbase/lib/hbase-protocol-1.3.1.jar:/usr/lib/hbase/lib/hbase-resource-bundle-1.3.1.jar:/usr/lib/hbase/lib/hbase-rest-1.3.1.jar:/usr/lib/hbase/lib/hbase-server-1.3.1.jar:/usr/lib/hbase/lib/hbase-server-1.3.1-tests.jar:/usr/lib/hbase/lib/hbase-shell-1.3.1.jar:/usr/lib/hbase/lib/hbase-thrift-1.3.1.jar:/usr/lib/hbase/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hbase/lib/httpclient-4.5.3.jar:/usr/lib/hbase/lib/httpcore-4.3.2.jar:/usr/lib/hbase/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hbase/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hbase/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hbase/lib/jackson-xc-1.9.13.jar:/usr/lib/hbase/lib/jamon-runtime-2.4.1.jar:/usr/lib/hbase/lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/lib/javax.inject-1.jar:/usr/lib/hbase/lib/java-xmlbuilder-0.4.jar:/usr/lib/hbase/lib/jaxb-api-2.2.2.jar:/usr/lib/hbase/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hbase/lib/jcodings-1.0.8.jar:/usr/lib/hbase/lib/jersey-client-1.9.jar:/usr/lib/hbase/lib/jersey-core-1.9.jar:/usr/lib/hbase/lib/jersey-guice-1.9.jar:/usr/lib/hbase/lib/jersey-json-1.9.jar:/usr/lib/hbase/lib/jersey-server-1.9.jar:/usr/lib/hbase/lib/jets3t-0.9.0.jar:/usr/lib/hbase/lib/jettison-1.3.3.jar:/usr/lib/hbase/lib/jetty-6.1.26.jar:/usr/lib/hbase/lib/jetty-sslengine-6.1.26.jar:/usr/lib/hbase/lib/jetty-util-6.1.26.jar:/usr/lib/hbase/lib/joni-2.1.2.jar:/usr/lib/hbase/lib/jruby-complete-1.6.8.jar:/usr/lib/hbase/lib/jsch-0.1.42.jar:/usr/lib/hbase/lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/lib/junit-4.12.jar:/usr/lib/hbase/lib/leveldbjni-all-1.8.jar:/usr/lib/hbase/lib/libthrift-0.9.3.jar:/usr/lib/hbase/lib/log4j-1.2.17.jar:/usr/lib/hbase/lib/metrics-core-2.2.0.jar:/usr/lib/hbase/lib/netty-all-4.0.23.Final.jar:/usr/lib/hbase/lib/paranamer-2.3.jar:/usr/lib/hbase/lib/protobuf-java-2.5.0.jar:/usr/lib/hbase/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/lib/servlet-api-2.5.jar:/usr/lib/hbase/lib/slf4j-api-1.6.1.jar:/usr/lib/hbase/lib/snappy-java-1.0.4.1.jar:/usr/lib/hbase/lib/spymemcached-2.11.6.jar:/usr/lib/hbase/lib/xmlenc-0.52.jar:/usr/lib/hbase/lib/xz-1.0.jar:/usr/lib/hbase/lib/zookeeper.jar:/etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/jets3t-0.9.0.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar:/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar:/usr/lib/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop/lib/junit-4.11.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/curator-client-2.7.1.jar:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/httpcore-4.4.4.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.0.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/gson-2.2.4.jar:/usr/lib/hadoop/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/httpclient-4.5.3.jar:/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.10.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop/lib/stax-api-1.0-2.jar:/usr/lib/hadoop/lib/commons-io-2.4.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/curator-framework-2.7.1.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/commons-lang-2.6.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/.//hadoop-datajoin.jar:/usr/lib/hadoop/.//hadoop-nfs-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-sls.jar:/usr/lib/hadoop/.//hadoop-aws.jar:/usr/lib/hadoop/.//hadoop-streaming.jar:/usr/lib/hadoop/.//hadoop-extras.jar:/usr/lib/hadoop/.//hadoop-openstack.jar:/usr/lib/hadoop/.//hadoop-rumen.jar:/usr/lib/hadoop/.//hadoop-streaming-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-azure.jar:/usr/lib/hadoop/.//hadoop-auth-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-archives.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-extras-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-gridmix.jar:/usr/lib/hadoop/.//hadoop-distcp-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-datajoin-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-aws-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-annotations-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-gridmix-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-archives-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-ant-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-ant.jar:/usr/lib/hadoop/.//hadoop-nfs.jar:/usr/lib/hadoop/.//hadoop-distcp.jar:/usr/lib/hadoop/.//hadoop-common-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-azure-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-rumen-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-sls-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-openstack-2.7.3-amzn-5.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-common-2.7.3-amzn-5-tests.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/lib/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/lib/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.7.3-amzn-5.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.3-amzn-5.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.7.3-amzn-5-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-yarn/lib/jetty-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/lib/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.4.jar:/usr/lib/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-yarn/lib/zookeeper-3.4.10-tests.jar:/usr/lib/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-yarn/lib/jettison-1.1.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-yarn/lib/activation-1.1.jar:/usr/lib/hadoop-yarn/lib/jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/guava-11.0.2.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-registry-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.7.3-amzn-5.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/lib/junit-4.11.jar:/usr/lib/hadoop-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/lib/guice-3.0.jar:/usr/lib/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/lib/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/lib/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/.//jetty-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/lib/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/lib/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-sls.jar:/usr/lib/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/lib/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/lib/hadoop-mapreduce/.//junit-4.11.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aws.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/lib/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/lib/hadoop-mapreduce/.//xz-1.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/lib/hadoop-mapreduce/.//aws-java-sdk-core-1.11.221.jar:/usr/lib/hadoop-mapreduce/.//httpcore-4.4.4.jar:/usr/lib/hadoop-mapreduce/.//jackson-annotations-2.6.7.jar:/usr/lib/hadoop-mapreduce/.//aws-java-sdk-kms-1.11.221.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure.jar:/usr/lib/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/lib/hadoop-mapreduce/.//jackson-dataformat-cbor-2.6.7.jar:/usr/lib/hadoop-mapreduce/.//asm-3.2.jar:/usr/lib/hadoop-mapreduce/.//hadoop-auth-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/lib/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-auth.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//jmespath-java-1.11.221.jar:/usr/lib/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/lib/hadoop-mapreduce/.//joda-time-2.8.1.jar:/usr/lib/hadoop-mapreduce/.//zookeeper-3.4.10.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/.//aws-java-sdk-s3-1.11.221.jar:/usr/lib/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/lib/hadoop-mapreduce/.//httpclient-4.5.3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/.//jackson-core-2.6.7.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/lib/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/lib/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aws-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/lib/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/lib/hadoop-mapreduce/.//azure-storage-2.0.0.jar:/usr/lib/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/lib/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//jackson-databind-2.6.7.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/lib/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/lib/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/lib/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/lib/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/lib/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-ant-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/lib/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/lib/hadoop-mapreduce/.//jettison-1.1.jar:/usr/lib/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/lib/hadoop-mapreduce/.//activation-1.1.jar:/usr/lib/hadoop-mapreduce/.//ion-java-1.0.2.jar:/usr/lib/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/lib/hadoop-mapreduce/.//jetty-util-6.1.26-emr.jar:/usr/lib/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/lib/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/lib/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/lib/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/lib/hadoop-mapreduce/.//hadoop-sls-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-openstack-2.7.3-amzn-5.jar:/usr/lib/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/lib/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/lib/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/lib/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.3-amzn-5-tests.jar::/etc/tez/conf:/usr/lib/tez/hadoop-shim-2.6-0.8.4.jar:/usr/lib/tez/hadoop-shim-0.8.4.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.8.4.jar:/usr/lib/tez/tez-runtime-internals-0.8.4.jar:/usr/lib/tez/tez-ext-service-tests-0.8.4.jar:/usr/lib/tez/tez-api-0.8.4.jar:/usr/lib/tez/tez-mapreduce-0.8.4.jar:/usr/lib/tez/tez-history-parser-0.8.4.jar:/usr/lib/tez/tez-examples-0.8.4.jar:/usr/lib/tez/tez-common-0.8.4.jar:/usr/lib/tez/tez-yarn-timeline-history-0.8.4.jar:/usr/lib/tez/tez-job-analyzer-0.8.4.jar:/usr/lib/tez/tez-javadoc-tools-0.8.4.jar:/usr/lib/tez/tez-runtime-library-0.8.4.jar:/usr/lib/tez/tez-dag-0.8.4.jar:/usr/lib/tez/tez-tests-0.8.4.jar:/usr/lib/tez/lib/jersey-json-1.9.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/RoaringBitmap-0.4.9.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/jetty-util-6.1.26.jar:/usr/lib/tez/lib/async-http-client-1.8.16.jar:/usr/lib/tez/lib/commons-collections-3.2.2.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-2.7.3-amzn-5.jar:/usr/lib/tez/lib/slf4j-api-1.7.10.jar:/usr/lib/tez/lib/jersey-client-1.9.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/servlet-api-2.5.jar:/usr/lib/tez/lib/commons-io-2.4.jar:/usr/lib/tez/lib/jetty-6.1.26.jar:/usr/lib/tez/lib/commons-math3-3.1.1.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-2.7.3-amzn-5.jar:/usr/lib/tez/lib/commons-codec-1.4.jar:/usr/lib/tez/lib/guava-11.0.2.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticache-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-health-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpleworkflow-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directconnect-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-directory-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-importexport-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ses-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elastictranscoder-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dynamodb-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-machinelearning-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-waf-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-config-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-datapipeline-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-acm-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workdocs-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatch-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-athena-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudtrail-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-devicefarm-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-redshift-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glue-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iam-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudformation-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsmv2-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lightsail-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-organizations-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sts-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecs-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lambda-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-migrationhub-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rds-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servermigration-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-core-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lex-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticsearch-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codepipeline-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-inspector-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-storagegateway-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kms-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codebuild-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-discovery-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sqs-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-shield-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-costandusagereport-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworks-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-kinesis-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opensdk-1.11.221.jar:/usr/share/aws/aws-java-sdk/jmespath-java-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudhsm-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ecr-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-greengrass-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitosync-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticbeanstalk-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-models-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-test-utils-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mobile-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacemeteringservice-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-rekognition-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-budgets-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplaceentitlement-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codestar-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-stepfunctions-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-clouddirectory-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ec2-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-sns-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-events-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-xray-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-autoscaling-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-gamelift-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codedeploy-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codecommit-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-emr-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudwatchmetrics-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-route53-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-code-generator-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-logs-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-marketplacecommerceanalytics-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-api-gateway-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-resourcegroupstaggingapi-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-batch-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-support-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidp-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-snowball-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancingv2-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-workspaces-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-codegen-maven-plugin-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudsearch-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-elasticloadbalancing-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-ssm-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dms-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-iot-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-simpledb-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-efs-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-opsworkscm-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-dax-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-glacier-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-lexmodelbuilding-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-polly-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cognitoidentity-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-applicationautoscaling-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-servicecatalog-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-mechanicalturkrequester-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-cloudfront-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-appstream-1.11.221.jar:/usr/share/aws/aws-java-sdk/aws-java-sdk-pinpoint-1.11.221.jar:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/bcpkix-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.20.0.jar:/usr/share/aws/emr/emrfs/lib/slf4j-api-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.11.129.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jcl-over-slf4j-1.7.21.jar:/usr/share/aws/emr/emrfs/lib/bcprov-jdk15on-1.51.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-1.0.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/etc/hadoop/conf
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.io.tmpdir=/tmp
2018-07-30 15:09:09,984 INFO  [main] zookeeper.ZooKeeper: Client 
environment:java.compiler=<NA>
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:os.name=Linux
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:os.arch=amd64
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:os.version=4.9.51-10.52.amzn1.x86_64
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:user.name=hbase
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:user.home=/var/lib/hbase
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Client 
environment:user.dir=/
2018-07-30 15:09:09,985 INFO  [main] zookeeper.ZooKeeper: Initiating client 
connection, connectString=ip-172-31-85-0.ec2.internal:2181 sessionTimeout=90000 
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@461c3709
2018-07-30 15:09:09,998 INFO  
[main-SendThread(ip-172-31-85-0.ec2.internal:2181)] zookeeper.ClientCnxn: 
Opening socket connection to server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181. Will not attempt to authenticate 
using SASL (unknown error)
2018-07-30 15:09:10,004 INFO  
[main-SendThread(ip-172-31-85-0.ec2.internal:2181)] zookeeper.ClientCnxn: 
Socket connection established to ip-172-31-85-0.ec2.internal/172.31.85.0:2181, 
initiating session
2018-07-30 15:09:10,012 INFO  
[main-SendThread(ip-172-31-85-0.ec2.internal:2181)] zookeeper.ClientCnxn: 
Session establishment complete on server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181, sessionid = 0x164eb7429950019, 
negotiated timeout = 40000
2018-07-30 15:09:10,050 INFO  [RpcServer.responder] ipc.RpcServer: 
RpcServer.responder: starting
2018-07-30 15:09:10,050 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: 
RpcServer.listener,port=16000: starting
2018-07-30 15:09:10,114 INFO  [main] mortbay.log: Logging to 
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2018-07-30 15:09:10,118 INFO  [main] http.HttpRequestLog: Http request log for 
http.requests.master is not defined
2018-07-30 15:09:10,127 INFO  [main] http.HttpServer: Added global filter 
'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2018-07-30 15:09:10,128 INFO  [main] http.HttpServer: Added global filter 
'clickjackingprevention' 
(class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)
2018-07-30 15:09:10,130 INFO  [main] http.HttpServer: Added filter 
static_user_filter 
(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) 
to context master
2018-07-30 15:09:10,130 INFO  [main] http.HttpServer: Added filter 
static_user_filter 
(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) 
to context static
2018-07-30 15:09:10,130 INFO  [main] http.HttpServer: Added filter 
static_user_filter 
(class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) 
to context logs
2018-07-30 15:09:10,150 INFO  [main] http.HttpServer: Jetty bound to port 16010
2018-07-30 15:09:10,150 INFO  [main] mortbay.log: jetty-6.1.26
2018-07-30 15:09:10,744 INFO  [main] mortbay.log: Started 
[email protected]:16010
2018-07-30 15:09:10,754 INFO  [main] master.HMaster: 
hbase.rootdir=s3://da-hbase/storage, hbase.cluster.distributed=true
2018-07-30 15:09:10,770 INFO  [main] master.HMaster: Adding backup master ZNode 
/hbase/backup-masters/ip-172-31-85-0.ec2.internal,16000,1532963345058
2018-07-30 15:09:10,874 INFO  [ip-172-31-85-0:16000.activeMasterManager] 
master.ActiveMasterManager: Deleting ZNode for 
/hbase/backup-masters/ip-172-31-85-0.ec2.internal,16000,1532963345058 from 
backup master directory
2018-07-30 15:09:10,884 INFO  [ip-172-31-85-0:16000.activeMasterManager] 
master.ActiveMasterManager: Registered Active 
Master=ip-172-31-85-0.ec2.internal,16000,1532963345058
2018-07-30 15:09:10,944 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7f04699d 
connecting to ZooKeeper ensemble=ip-172-31-85-0.ec2.internal:2181
2018-07-30 15:09:10,944 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] zookeeper.ZooKeeper: 
Initiating client connection, connectString=ip-172-31-85-0.ec2.internal:2181 
sessionTimeout=90000 
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@29bc3c42
2018-07-30 15:09:10,946 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000-SendThread(ip-172-31-85-0.ec2.internal:2181)]
 zookeeper.ClientCnxn: Opening socket connection to server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181. Will not attempt to authenticate 
using SASL (unknown error)
2018-07-30 15:09:10,948 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000-SendThread(ip-172-31-85-0.ec2.internal:2181)]
 zookeeper.ClientCnxn: Socket connection established to 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181, initiating session
2018-07-30 15:09:10,957 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000-SendThread(ip-172-31-85-0.ec2.internal:2181)]
 zookeeper.ClientCnxn: Session establishment complete on server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181, sessionid = 0x164eb742995001a, 
negotiated timeout = 40000
2018-07-30 15:09:10,958 INFO  
[master/ip-172-31-85-0.ec2.internal/172.31.85.0:16000] 
client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
2018-07-30 15:09:11,483 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:09:21,683 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:10:04,756 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:11:16,923 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:13:00,016 ERROR [ip-172-31-85-0:16000.activeMasterManager] 
consistency.ConsistencyCheckerS3FileSystem: No s3 object for metadata item 
/da-hbase/storage/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-07-30 15:15:00,585 INFO  [829985002@qtp-393594087-0] 
zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x70f7fa6d 
connecting to ZooKeeper ensemble=ip-172-31-85-0.ec2.internal:2181
2018-07-30 15:15:00,585 INFO  [829985002@qtp-393594087-0] zookeeper.ZooKeeper: 
Initiating client connection, connectString=ip-172-31-85-0.ec2.internal:2181 
sessionTimeout=90000 
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@79dcf2a1
2018-07-30 15:15:00,590 INFO  
[829985002@qtp-393594087-0-SendThread(ip-172-31-85-0.ec2.internal:2181)] 
zookeeper.ClientCnxn: Opening socket connection to server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181. Will not attempt to authenticate 
using SASL (unknown error)
2018-07-30 15:15:00,590 INFO  
[829985002@qtp-393594087-0-SendThread(ip-172-31-85-0.ec2.internal:2181)] 
zookeeper.ClientCnxn: Socket connection established to 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181, initiating session
2018-07-30 15:15:00,592 INFO  
[829985002@qtp-393594087-0-SendThread(ip-172-31-85-0.ec2.internal:2181)] 
zookeeper.ClientCnxn: Session establishment complete on server 
ip-172-31-85-0.ec2.internal/172.31.85.0:2181, sessionid = 0x164eb742995001b, 
negotiated timeout = 40000
2018-07-30 15:15:00,592 INFO  [829985002@qtp-393594087-0] 
client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
2018-07-30 15:15:07,997 INFO  [1834223683@qtp-393594087-3] logs: Aliases are 
enabled
2018-07-30 15:15:39,055 INFO  [829985002@qtp-393594087-0] 
client.RpcRetryingCaller: Call exception, tries=10, retries=350, started=38460 
ms ago, cancelled=false, msg=com.google.protobuf.ServiceException: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException):
 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not 
running yet
        at 
org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2500)
        at 
org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:976)
        at 
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:61715)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)

El 30 jul 2018, a las 9:02, <[email protected]> <[email protected]> escribió:

Hi Moisés,

If I have understood right you have been able to deploy Kylin on EMR successfully . However you lose metadata when you terminate the cluster. is it right?

Have you tried to restore Kylin metadata backup after cluster re-creation? Moreover, do you enable all HBase tables after cluster re-creation?

We successfully deployed Kylin on EMR using S3 as storage for HBase and Hive. But our configuration differ by 2 points:
·         We use EMRFS https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-fs.html
o   {
o                  "Classification": "emrfs-site",
o                  "Properties": {
o                                  "fs.s3.consistent.retryPeriodSeconds": "10",
o                                  "fs.s3.consistent": "true",
o                                  "fs.s3.consistent.retryCount": "5",
o                                  "fs.s3.consistent.metadata.tableName": "EmrFSMetadata"
o                  },
o                  "Configurations": []
o   }
·         We deployed Kylin on an EC2 machine separated from the cluster.

I hope this helps you.

Roberto Tardío

From: Moisés Català [mailto:[email protected]]
Sent: sábado, 28 de julio de 2018 16:17
To: [email protected]
Subject: Kylin with S3, cubes tables get in transition when new cluster booted

Hi all,

I’ve followed carefully the instructions provided in http://kylin.apache.org/docs23/install/kylin_aws_emr.html

My idea is to use s3 as the storage for Hbase, I have configured the cluster following the instructions but I get that tables that contain cube definition keep "on transition" when deploying new cluster and Kylie metadata seems outdated...

These are the steps I follow to create the cluster

Cluster creation command:

aws emr create-cluster \
--applications Name=Hadoop Name=Hue Name=Spark Name=Zeppelin Name=Ganglia Name=Hive Name=Hbase Name=HCatalog Name=Tez \
--tags 'hive=' 'spark=' 'zeppelin=' \
--ec2-attributes 'file://../config/ec2-attributes.json' \
--release-label emr-5.16.0 \
--log-uri 's3n://sns-da-logs/' \
--instance-groups 'file://../config/instance-hive-datawarehouse.json' \
--configurations 'file://../config/hive-hbase-s3.json' \
--auto-scaling-role EMR_AutoScaling_DefaultRole \
--ebs-root-volume-size 10 \
--service-role EMR_DefaultRole \
--enable-debugging \
--name 'hbase-hive-datawarehouse' \
--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
--region us-east-1

My configuration hive-hbase-s3.json:

[
{
"Classification": "hive-site",
"Configurations": [],
"Properties": {
"hive.metastore.warehouse.dir": "s3://xxxxxxxx-datawarehouse/hive.db",
"javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",
"javax.jdo.option.ConnectionPassword": “xxxxx",
"javax.jdo.option.ConnectionURL": "jdbc:mysql://xxxxxx:3306/hive_metastore?createDatabaseIfNotExist=true",
"javax.jdo.option.ConnectionUserName": “xxxx"
}
},
{
"Classification": "hbase",
"Configurations": [],
"Properties": {
"hbase.emr.storageMode": "s3"
}
},
{
"Classification": "hbase-site",
"Configurations": [],
"Properties": {
"hbase.rpc.timeout": "3600000",
"hbase.rootdir": "s3://xxxxxx-hbase/"
}
},
{
"Classification": "core-site",
"Properties": {
"io.file.buffer.size": "65536"
}
},
{
"Classification": "mapred-site",
"Properties": {
"mapred.map.tasks.speculative.execution": "false",
"mapred.reduce.tasks.speculative.execution": "false",
"mapreduce.map.speculative": "false",
"mapreduce.reduce.speculative": "false"

}
}
]

When I shut down the cluster I perform these commands:

../kylin_home/bin/kylin.sh stop

#Before you shutdown/restart the cluster, you must backup the “/kylin” data on HDFS to S3 with S3DistCp,

aws s3 rm s3://xxxxxx-config/metadata/kylin/*
s3-dist-cp --src="">hdfs:///kylin --dest=s3://xxxxxx-config/metadata/kylin

bash /usr/lib/hbase/bin/disable_all_tables.sh

Please, could you be so kind to indicate me what am I missing

Thanks in advance

Re: Kylin with S3, cubes tables get in transition when new cluster booted

Reply via email to