[jira] [Commented] (HBASE-27544) RegionServer JVM crash when the RPC request size is too big

2022-12-22 Thread Yiran Wu (Jira) 


[ 
https://issues.apache.org/jira/browse/HBASE-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651312#comment-17651312
 ] 

Yiran Wu commented on HBASE-27544:
--

[~Xiaolin Ha] Thanks , It seems to be related to HBASE-25997. Let me try .

> RegionServer JVM crash when the RPC request size is too big
> ---
>
> Key: HBASE-27544
> URL: https://issues.apache.org/jira/browse/HBASE-27544
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: Yiran Wu
>Priority: Major
>
> In our cluster,  JVM crash when the request size is too big
> hs_err.log
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f6c4bbb7b31, pid=5619, tid=0x7f3dc57b4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 
> 1.8.0_181-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode 
> linux-amd64 )
> # Problematic frame:
> # C  [libc.so.6+0x15bb31]  __memmove_ssse3_back+0x1ba1
> #
> # Core dump written. Default location: /home/user/core or core.5619 (max size 
> 1048576 kB). To ensure a full core dump, try "ulimit -c unlimited" before 
> starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> Stack: [0x7f3dc56b4000,0x7f3dc57b5000],  sp=0x7f3dc57b2d48,  free 
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native 
> code)
> C  [libc.so.6+0x15bb31]  __memmove_ssse3_back+0x1ba1
> J 2301  sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V 
> (0 bytes) @ 0x7f6c35a3ae21 [0x7f6c35a3ad40+0xe1]
> j  
> org.apache.hadoop.hbase.util.UnsafeAccess.unsafeCopy(Ljava/lang/Object;JLjava/lang/Object;JJ)V+36
> j  
> org.apache.hadoop.hbase.util.UnsafeAccess.copy(Ljava/nio/ByteBuffer;I[BII)V+69
> j  
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V+39
> j  
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I+31
> j  
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> j  org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B+1
> j  
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j  
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j  
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j  org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j  
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> j  
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;+2
> J 19274 C2 
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
>  (910 bytes) @ 0x7f6c386ed4e4 [0x7f6c386eb7a0+0x1d44]
> J 32557 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;JLorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
>  (1046 bytes) @ 0x7f6c39d9e494 [0x7f6c39d9dcc0+0x7d4]
> J 29517 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
>  (901 bytes) @ 0x7f6c39c25898 [0x7f6c39c24da0+0xaf8]
> J 31074 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(Lorg/apache/hbase/thirdparty/com/google/protobuf/RpcController;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MultiRequest;)Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MultiResponse;
>  (1119 bytes) @ 0x7f6c39e7dcd4 [0x7f6c39e7db20+0x1b4]
> J 28404 C2 
>

[jira] [Commented] (HBASE-27544) RegionServer JVM crash when the RPC request size is too big

2022-12-22 Thread Xiaolin Ha (Jira) 


[ 
https://issues.apache.org/jira/browse/HBASE-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651184#comment-17651184
 ] 

Xiaolin Ha commented on HBASE-27544:


There are similar and duplicate issues, please see HBASE-26170，HBASE-25997.

> RegionServer JVM crash when the RPC request size is too big
> ---
>
> Key: HBASE-27544
> URL: https://issues.apache.org/jira/browse/HBASE-27544
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.3.4
>Reporter: Yiran Wu
>Priority: Major
>
> In our cluster,  JVM crash when the request size is too big
> hs_err.log
> {code:java}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7f6c4bbb7b31, pid=5619, tid=0x7f3dc57b4700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 
> 1.8.0_181-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode 
> linux-amd64 )
> # Problematic frame:
> # C  [libc.so.6+0x15bb31]  __memmove_ssse3_back+0x1ba1
> #
> # Core dump written. Default location: /home/user/core or core.5619 (max size 
> 1048576 kB). To ensure a full core dump, try "ulimit -c unlimited" before 
> starting Java again
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> #
> Stack: [0x7f3dc56b4000,0x7f3dc57b5000],  sp=0x7f3dc57b2d48,  free 
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native 
> code)
> C  [libc.so.6+0x15bb31]  __memmove_ssse3_back+0x1ba1
> J 2301  sun.misc.Unsafe.copyMemory(Ljava/lang/Object;JLjava/lang/Object;JJ)V 
> (0 bytes) @ 0x7f6c35a3ae21 [0x7f6c35a3ad40+0xe1]
> j  
> org.apache.hadoop.hbase.util.UnsafeAccess.unsafeCopy(Ljava/lang/Object;JLjava/lang/Object;JJ)V+36
> j  
> org.apache.hadoop.hbase.util.UnsafeAccess.copy(Ljava/nio/ByteBuffer;I[BII)V+69
> j  
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V+39
> j  
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I+31
> j  
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> j  org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B+1
> j  
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j  
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j  
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j  org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j  
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> j  
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;+2
> J 19274 C2 
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
>  (910 bytes) @ 0x7f6c386ed4e4 [0x7f6c386eb7a0+0x1d44]
> J 32557 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;JLorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
>  (1046 bytes) @ 0x7f6c39d9e494 [0x7f6c39d9dcc0+0x7d4]
> J 29517 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
>  (901 bytes) @ 0x7f6c39c25898 [0x7f6c39c24da0+0xaf8]
> J 31074 C2 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(Lorg/apache/hbase/thirdparty/com/google/protobuf/RpcController;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MultiRequest;)Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MultiResponse;
>  (1119 bytes) @ 0x7f6c39e7dcd4 [0x7f6c39e7db20+0x1b4]
> J 28404 C2 
>