[
https://issues.apache.org/jira/browse/HBASE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang resolved HBASE-26024.
-------------------------------
Resolution: Duplicate
> Region server JVM Crash - A fatal error has been detected by the Java Runtime
> Environment
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-26024
> URL: https://issues.apache.org/jira/browse/HBASE-26024
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.1.9
> Reporter: Mohamed Mohideen Meeran
> Priority: Major
> Attachments: error.log
>
>
> Our production Region servers JVM crashed with the following error logs.
>
> Register to memory mapping:
> RAX=0x000000002eea8f42 is an unknown value
> RBX=0x00007f1f8c7900d6 is an unknown value
> RCX=0x0000000000000021 is an unknown value
> RDX=0x0000000000000000 is an unknown value
> RSP=0x00007f1fe3092200 is pointing into the stack for thread:
> 0x00007f29775fb000
> RBP=0x00007f1fe3092200 is pointing into the stack for thread:
> 0x00007f29775fb000
> RSI=0x00007f1f8c7900cc is an unknown value
> RDI=0x000000002eea8f38 is an unknown value
> R8 =0x00007f28e14a3a38 is an oop
> java.nio.DirectByteBuffer
> - klass: 'java/nio/DirectByteBuffer'
> R9 =0x00007f1f8c790094 is an unknown value
> R10=0x00007f2965053400 is at begin+0 in a stub
> StubRoutines::unsafe_arraycopy [0x00007f2965053400, 0x00007f296505343b[ (59
> bytes)
> R11=0x00007f28e14a3a38 is an oop
> java.nio.DirectByteBuffer
> - klass: 'java/nio/DirectByteBuffer'
> R12=
> [error occurred during error reporting (printing register info), id 0xb]
>
> Stack: [0x00007f1fe2f93000,0x00007f1fe3094000], sp=0x00007f1fe3092200, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jshort_disjoint_arraycopy
> J 18388 C2
> org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V
> (53 bytes) @ 0x00007f2967fa0ea2 [0x00007f2967fa0d40+0x162]
> J 11722 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V
> (75 bytes) @ 0x00007f29670aa0fc [0x00007f29670a9fa0+0x15c]
> J 13251 C2
> org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I
> (21 bytes) @ 0x00007f2965cbe87c [0x00007f2965cbe820+0x5c]
> J 8703 C2
> org.apache.hadoop.hbase.KeyValueUtil.oswrite(Lorg/apache/hadoop/hbase/Cell;Ljava/io/OutputStream;Z)I
> (259 bytes) @ 0x00007f296684a2d4 [0x00007f296684a140+0x194]
> J 15474 C2
> org.apache.hadoop.hbase.ipc.CellBlockBuilder.buildCellBlockStream(Lorg/apache/hadoop/hbase/codec/Codec;Lorg/apache/hadoop/io/compress/CompressionCodec;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/io/ByteBufferPool;)Lorg/apache/hadoop/hbase/io/ByteBufferListOutputStream;
> (75 bytes) @ 0x00007f29675f9dc8 [0x00007f29675f7c80+0x2148]
> J 14260 C2
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(Lorg/apache/hbase/thirdparty/com/google/protobuf/Message;Lorg/apache/hadoop/hbase/CellScanner;Ljava/lang/Throwable;Ljava/lang/String;)V
> (408 bytes) @ 0x00007f29678ad11c [0x00007f29678acec0+0x25c]
> J 14732 C2 org.apache.hadoop.hbase.ipc.CallRunner.run()V (1376 bytes) @
> 0x00007f296797f690 [0x00007f296797e6a0+0xff0]
> J 14293 C2
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(Lorg/apache/hadoop/hbase/ipc/CallRunner;)V
> (268 bytes) @ 0x00007f29667b7464 [0x00007f29667b72e0+0x184]
> J 17796% C1 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run()V (72 bytes)
> @ 0x00007f2967c9cbe4 [0x00007f2967c9caa0+0x144]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x65ebbb] JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x108b
> V [libjvm.so+0x65ffd7] JavaCalls::call_virtual(JavaValue*, KlassHandle,
> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x2f7
> V [libjvm.so+0x660497] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x47
> V [libjvm.so+0x6ada71] thread_entry(JavaThread*, Thread*)+0x91
> V [libjvm.so+0x9f24f1] JavaThread::thread_main_inner()+0xf1
> V [libjvm.so+0x9f26d8] JavaThread::run()+0x1b8
> V [libjvm.so+0x8af502] java_start(Thread*)+0x122
> C [libpthread.so.0+0x7dc5] start_thread+0xc5
>
> we used the workaround by switching *hbase.rpc.server.impl* back to
> SimpleRpcServer as mentioned in the following JIRA
> https://issues.apache.org/jira/browse/HBASE-22539?focusedCommentId=16855688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16855688
> Also, attached the error logs during JVM crash. Any help?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
