[ https://issues.apache.org/jira/browse/AMBARI-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433149#comment-15433149 ]
Dmytro Grinenko commented on AMBARI-18238: ------------------------------------------ commited to trunk bdae701..e671d0f trunk -> trunk > HBase Master doesn't start after upgrading from HDP 2.3.6 using Ambari > ---------------------------------------------------------------------- > > Key: AMBARI-18238 > URL: https://issues.apache.org/jira/browse/AMBARI-18238 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: trunk > Reporter: Dmytro Grinenko > Priority: Critical > Fix For: trunk > > Attachments: AMBARI-18238.patch > > > After upgrading the HBase server doesn't start throws below error. > {code} > 016-08-21 05:49:27,298 INFO [main] util.ServerCommandLine: > env:SERVER_GC_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps > -Xloggc:/var/log/hbase/gc.log-201608210549 > 2016-08-21 05:49:27,298 INFO [main] util.ServerCommandLine: > env:HADOOP_CONF=/usr/hdp/2.5.0.0-1232/hadoop/conf > 2016-08-21 05:49:27,298 INFO [main] util.ServerCommandLine: > env:HOSTNAME=hcube1-1n03.eng.hortonworks.com > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:QTDIR=/usr/lib64/qt-3.3 > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:HBASE_THRIFT_OPTS= > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:HBASE_HOME=/usr/hdp/current/hbase-master/bin/.. > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:QTLIB=/usr/lib64/qt-3.3/lib > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:HOME=/home/hbase > 2016-08-21 05:49:27,299 INFO [main] util.ServerCommandLine: > env:MALLOC_ARENA_MAX=4 > 2016-08-21 05:49:27,301 INFO [main] util.ServerCommandLine: vmName=Java > HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.60-b23 > 2016-08-21 05:49:27,302 INFO [main] util.ServerCommandLine: > vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, > -Dhdp.version=2.5.0.0-1232, -XX:+UseConcMarkSweepGC, > -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log, > -Djava.security.auth.login.config=/usr/hdp/current/hbase-master/conf/hbase_client_jaas.conf, > -Djava.io.tmpdir=/tmp, -verbose:gc, -XX:+PrintGCDetails, > -XX:+PrintGCDateStamps, -Xloggc:/var/log/hbase/gc.log-201608210549, > -Xmx2048m, > -Djava.security.auth.login.config=/usr/hdp/current/hbase-master/conf/hbase_master_jaas.conf, > -Dhbase.log.dir=/var/log/hbase, > -Dhbase.log.file=hbase-hbase-master-hcube1-1n03.eng.hortonworks.com.log, > -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.., -Dhbase.id.str=hbase, > -Dhbase.root.logger=INFO,RFA, > -Djava.library.path=:/usr/hdp/2.5.0.0-1232/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.5.0.0-1232/hadoop/lib/native, > -Dhbase.security.logger=INFO,RFAS] > 2016-08-21 05:49:27,715 INFO [main] regionserver.RSRpcServices: > master/hcube1-1n03.eng.hortonworks.com/172.18.129.3:16000 server-side > HConnection retries=350 > 2016-08-21 05:49:27,892 INFO [main] ipc.SimpleRpcScheduler: Using fifo as > user call queue, count=12 > 2016-08-21 05:49:27,896 INFO [main] ipc.PhoenixRpcSchedulerFactory: Using > custom Phoenix Index RPC Handling with index rpc priority 1000 and metadata > rpc priority 2000 > 2016-08-21 05:49:27,919 INFO [main] ipc.RpcServer: > master/hcube1-1n03.eng.hortonworks.com/172.18.129.3:16000: started 10 > reader(s). > 2016-08-21 05:49:28,027 INFO [main] impl.MetricsConfig: loaded properties > from hadoop-metrics2-hbase.properties > 2016-08-21 05:49:28,063 INFO [main] timeline.HadoopTimelineMetricsSink: > Initializing Timeline metrics sink. > 2016-08-21 05:49:28,063 INFO [main] timeline.HadoopTimelineMetricsSink: > Identified hostname = hcube1-1n03.eng.hortonworks.com, serviceName = hbase > 2016-08-21 05:49:28,174 INFO [main] timeline.HadoopTimelineMetricsSink: > Collector Uri: > http://hcube1-2n01.eng.hortonworks.com:6188/ws/v1/timeline/metrics > 2016-08-21 05:49:28,189 INFO [main] impl.MetricsSinkAdapter: Sink timeline > started > 2016-08-21 05:49:28,278 INFO [main] impl.MetricsSystemImpl: Scheduled > snapshot period at 10 second(s). > 2016-08-21 05:49:28,278 INFO [main] impl.MetricsSystemImpl: HBase metrics > system started > 2016-08-21 05:49:28,649 INFO [main] security.UserGroupInformation: Login > successful for user > hbase/hcube1-1n03.eng.hortonworks....@hwx.stanleyhotel.com using keytab file > /etc/security/keytabs/hbase.service.keytab > 2016-08-21 05:49:28,655 INFO [main] hfile.CacheConfig: Allocating > LruBlockCache size=791.90 MB, blockSize=64 KB > 2016-08-21 05:49:28,663 WARN [main] hfile.CacheConfig: Configuration > 'hbase.bucketcache.percentage.in.combinedcache' is no longer respected. See > comments in http://hbase.apache.org/book.html#_changes_of_note > 2016-08-21 05:49:28,674 INFO [main] util.ByteBufferArray: Allocating buffers > total=10 GB, sizePerBuffer=4 MB, count=2560, direct=true > 2016-08-21 05:49:30,320 ERROR [main] master.HMasterCommandLine: Master exiting > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2515) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2529) > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) > at > org.apache.hadoop.hbase.util.ByteBufferArray.<init>(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.<init>(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.<init>(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.<init>(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:564) > at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:411) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2510) > ... 5 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)