[ https://issues.apache.org/jira/browse/HBASE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368397#comment-17368397 ]
Josh Elser commented on HBASE-26024: ------------------------------------ Any chance you have access to an hbase 2.2 or 2.3 cluster in which you can see if this same problem happens there? There have been some fixes I recall seeing around Netty and direct bytebuffers in the recent past. > Region server JVM Crash - A fatal error has been detected by the Java Runtime > Environment > ----------------------------------------------------------------------------------------- > > Key: HBASE-26024 > URL: https://issues.apache.org/jira/browse/HBASE-26024 > Project: HBase > Issue Type: Bug > Affects Versions: 2.1.9 > Reporter: Mohamed Mohideen Meeran > Priority: Major > Attachments: error.log > > > Our production Region servers JVM crashed with the following error logs. > > Register to memory mapping: > RAX=0x000000002eea8f42 is an unknown value > RBX=0x00007f1f8c7900d6 is an unknown value > RCX=0x0000000000000021 is an unknown value > RDX=0x0000000000000000 is an unknown value > RSP=0x00007f1fe3092200 is pointing into the stack for thread: > 0x00007f29775fb000 > RBP=0x00007f1fe3092200 is pointing into the stack for thread: > 0x00007f29775fb000 > RSI=0x00007f1f8c7900cc is an unknown value > RDI=0x000000002eea8f38 is an unknown value > R8 =0x00007f28e14a3a38 is an oop > java.nio.DirectByteBuffer > - klass: 'java/nio/DirectByteBuffer' > R9 =0x00007f1f8c790094 is an unknown value > R10=0x00007f2965053400 is at begin+0 in a stub > StubRoutines::unsafe_arraycopy [0x00007f2965053400, 0x00007f296505343b[ (59 > bytes) > R11=0x00007f28e14a3a38 is an oop > java.nio.DirectByteBuffer > - klass: 'java/nio/DirectByteBuffer' > R12= > [error occurred during error reporting (printing register info), id 0xb] > > Stack: [0x00007f1fe2f93000,0x00007f1fe3094000], sp=0x00007f1fe3092200, free > space=1020k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > v ~StubRoutines::jshort_disjoint_arraycopy > J 18388 C2 > org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V > (53 bytes) @ 0x00007f2967fa0ea2 [0x00007f2967fa0d40+0x162] > J 11722 C2 > org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V > (75 bytes) @ 0x00007f29670aa0fc [0x00007f29670a9fa0+0x15c] > J 13251 C2 > org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I > (21 bytes) @ 0x00007f2965cbe87c [0x00007f2965cbe820+0x5c] > J 8703 C2 > org.apache.hadoop.hbase.KeyValueUtil.oswrite(Lorg/apache/hadoop/hbase/Cell;Ljava/io/OutputStream;Z)I > (259 bytes) @ 0x00007f296684a2d4 [0x00007f296684a140+0x194] > J 15474 C2 > org.apache.hadoop.hbase.ipc.CellBlockBuilder.buildCellBlockStream(Lorg/apache/hadoop/hbase/codec/Codec;Lorg/apache/hadoop/io/compress/CompressionCodec;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/io/ByteBufferPool;)Lorg/apache/hadoop/hbase/io/ByteBufferListOutputStream; > (75 bytes) @ 0x00007f29675f9dc8 [0x00007f29675f7c80+0x2148] > J 14260 C2 > org.apache.hadoop.hbase.ipc.ServerCall.setResponse(Lorg/apache/hbase/thirdparty/com/google/protobuf/Message;Lorg/apache/hadoop/hbase/CellScanner;Ljava/lang/Throwable;Ljava/lang/String;)V > (408 bytes) @ 0x00007f29678ad11c [0x00007f29678acec0+0x25c] > J 14732 C2 org.apache.hadoop.hbase.ipc.CallRunner.run()V (1376 bytes) @ > 0x00007f296797f690 [0x00007f296797e6a0+0xff0] > J 14293 C2 > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(Lorg/apache/hadoop/hbase/ipc/CallRunner;)V > (268 bytes) @ 0x00007f29667b7464 [0x00007f29667b72e0+0x184] > J 17796% C1 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run()V (72 bytes) > @ 0x00007f2967c9cbe4 [0x00007f2967c9caa0+0x144] > v ~StubRoutines::call_stub > V [libjvm.so+0x65ebbb] JavaCalls::call_helper(JavaValue*, methodHandle*, > JavaCallArguments*, Thread*)+0x108b > V [libjvm.so+0x65ffd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, > Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x2f7 > V [libjvm.so+0x660497] JavaCalls::call_virtual(JavaValue*, Handle, > KlassHandle, Symbol*, Symbol*, Thread*)+0x47 > V [libjvm.so+0x6ada71] thread_entry(JavaThread*, Thread*)+0x91 > V [libjvm.so+0x9f24f1] JavaThread::thread_main_inner()+0xf1 > V [libjvm.so+0x9f26d8] JavaThread::run()+0x1b8 > V [libjvm.so+0x8af502] java_start(Thread*)+0x122 > C [libpthread.so.0+0x7dc5] start_thread+0xc5 > > we used the workaround by switching *hbase.rpc.server.impl* back to > SimpleRpcServer as mentioned in the following JIRA > https://issues.apache.org/jira/browse/HBASE-22539?focusedCommentId=16855688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16855688 > Also, attached the error logs during JVM crash. Any help? -- This message was sent by Atlassian Jira (v8.3.4#803005)