[jira] [Updated] (AMBARI-15355) After EU from 2.2 -> 2.3 HBase region server started failed with memory error
[ https://issues.apache.org/jira/browse/AMBARI-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lysnichenko updated AMBARI-15355: Fix Version/s: 2.2.2 > After EU from 2.2 -> 2.3 HBase region server started failed with memory error > - > > Key: AMBARI-15355 > URL: https://issues.apache.org/jira/browse/AMBARI-15355 > Project: Ambari > Issue Type: Bug > Components: ambari-server >Affects Versions: 2.2.2 >Reporter: Dmitry Lysnichenko >Assignee: Dmitry Lysnichenko > Fix For: 2.2.2 > > Attachments: AMBARI-15355.patch > > > On one of the clusters i did EU from 2.2.x to 2.3.x. > During upgrade there were problems with HBase service checks for region > servers and thus upgrade is paused. > Region server start is failing with error > {code} > 2016-03-03 19:55:31,203 ERROR [regionserver:16020] > regionserver.HRegionServer: Failed init > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:31,206 FATAL [regionserver:16020] > regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort > itself immediately > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:35,138 INFO [main] zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT > {code} > This was seen on the following cluster: > https://s.c:8443/#/main/services/HBASE/configs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-15355) After EU from 2.2 -> 2.3 HBase region server started failed with memory error
[ https://issues.apache.org/jira/browse/AMBARI-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lysnichenko updated AMBARI-15355: Resolution: Fixed Status: Resolved (was: Patch Available) To https://git-wip-us.apache.org/repos/asf/ambari.git dc5ddfc..4051f69 branch-2.2 -> branch-2.2 456b451..28430f3 trunk -> trunk > After EU from 2.2 -> 2.3 HBase region server started failed with memory error > - > > Key: AMBARI-15355 > URL: https://issues.apache.org/jira/browse/AMBARI-15355 > Project: Ambari > Issue Type: Bug > Components: ambari-server >Reporter: Dmitry Lysnichenko >Assignee: Dmitry Lysnichenko > Attachments: AMBARI-15355.patch > > > On one of the clusters i did EU from 2.2.x to 2.3.x. > During upgrade there were problems with HBase service checks for region > servers and thus upgrade is paused. > Region server start is failing with error > {code} > 2016-03-03 19:55:31,203 ERROR [regionserver:16020] > regionserver.HRegionServer: Failed init > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:31,206 FATAL [regionserver:16020] > regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort > itself immediately > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:35,138 INFO [main] zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT > {code} > This was seen on the following cluster: > https://s.c:8443/#/main/services/HBASE/configs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AMBARI-15355) After EU from 2.2 -> 2.3 HBase region server started failed with memory error
[ https://issues.apache.org/jira/browse/AMBARI-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Lysnichenko updated AMBARI-15355: Attachment: AMBARI-15355.patch > After EU from 2.2 -> 2.3 HBase region server started failed with memory error > - > > Key: AMBARI-15355 > URL: https://issues.apache.org/jira/browse/AMBARI-15355 > Project: Ambari > Issue Type: Bug > Components: ambari-server >Reporter: Dmitry Lysnichenko >Assignee: Dmitry Lysnichenko > Attachments: AMBARI-15355.patch > > > On one of the clusters i did EU from 2.2.x to 2.3.x. > During upgrade there were problems with HBase service checks for region > servers and thus upgrade is paused. > Region server start is failing with error > {code} > 2016-03-03 19:55:31,203 ERROR [regionserver:16020] > regionserver.HRegionServer: Failed init > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:31,206 FATAL [regionserver:16020] > regionserver.RSRpcServices: Run out of memory; RSRpcServices will abort > itself immediately > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.apache.hadoop.hbase.util.ByteBufferArray.(ByteBufferArray.java:65) > at > org.apache.hadoop.hbase.io.hfile.bucket.ByteBufferIOEngine.(ByteBufferIOEngine.java:47) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getIOEngineFromName(BucketCache.java:307) > at > org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.(BucketCache.java:217) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.getBucketCache(CacheConfig.java:614) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.getL2(CacheConfig.java:553) > at > org.apache.hadoop.hbase.io.hfile.CacheConfig.instantiateBlockCache(CacheConfig.java:637) > at org.apache.hadoop.hbase.io.hfile.CacheConfig.(CacheConfig.java:231) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1361) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 19:55:35,138 INFO [main] zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT > {code} > This was seen on the following cluster: > https://s.c:8443/#/main/services/HBASE/configs -- This message was sent by Atlassian JIRA (v6.3.4#6332)