[jira] [Commented] (HBASE-24623) SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
[
https://issues.apache.org/jira/browse/HBASE-24623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146019#comment-17146019
]
Michael Stack commented on HBASE-24623:
---
No offheap BC, no offheap write path, all defaults. I do notice though that
hbase-2.3.0 is first release with ByteBufferAllocator vs ByteBufferPool; the
former does refcounting where the latter did not.
> SIGSEGV v ~StubRoutines::jbyte_disjoint_arraycopy
> --
>
> Key: HBASE-24623
> URL: https://issues.apache.org/jira/browse/HBASE-24623
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
>
> In testing, 1% of a decent cluster went down with this seg fault in the vm:
> {code}
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x7f6659052410, pid=37208, tid=0x7f3c89453700
> #
> # JRE version: OpenJDK Runtime Environment (8.0_232-b09) (build 1.8.0_232-b09)
> # Java VM: OpenJDK 64-Bit Server VM (25.232-b09 mixed mode linux-amd64 )
> # Problematic frame:
> # v ~StubRoutines::jbyte_disjoint_arraycopy
> {code}
> Looking in the hs_err log, the crash happens in the same area. Here are a few
> of the stack traces:
> {code}
> Stack: [0x7f3c89353000,0x7f3c89454000], sp=0x7f3c89452110, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 17674 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f665af000d1 [0x7f665aefffe0+0xf1]
> J 17732 C1
> org.apache.hadoop.hbase.CellUtil.copyQualifierTo(Lorg/apache/hadoop/hbase/Cell;[BI)I
> (59 bytes) @ 0x7f665bc440dc [0x7f665bc43b80+0x55c]
> j
> org.apache.hadoop.hbase.CellUtil.cloneQualifier(Lorg/apache/hadoop/hbase/Cell;)[B+12
> J 22278 C2 org.apache.hadoop.hbase.ByteBufferKeyValue.getQualifierArray()[B
> (5 bytes) @ 0x7f6659bd4784 [0x7f6659bd4760+0x24]
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;Ljava/util/function/Function;)Ljava/lang/String;+97
> j
> org.apache.hadoop.hbase.CellUtil.getCellKeyAsString(Lorg/apache/hadoop/hbase/Cell;)Ljava/lang/String;+6
> j
> org.apache.hadoop.hbase.CellUtil.toString(Lorg/apache/hadoop/hbase/Cell;Z)Ljava/lang/String;+16
> j org.apache.hadoop.hbase.ByteBufferKeyValue.toString()Ljava/lang/String;+2
> j
> org.apache.hadoop.hbase.client.Mutation.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Mutation;+28
> J 22605 C2
> org.apache.hadoop.hbase.client.Put.add(Lorg/apache/hadoop/hbase/Cell;)Lorg/apache/hadoop/hbase/client/Put;
> (8 bytes) @ 0x7f665a982a04 [0x7f665a9829e0+0x24]
> J 22112 C2
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toPut(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$MutationProto;Lorg/apache/hadoop/hbase/CellScanner;)Lorg/apache/hadoop/hbase/client/Put;
> (910 bytes) @ 0x7f665c706700 [0x7f665c706000+0x700]
> J 24084 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Ljava/util/List;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;Z)V
> (646 bytes) @ 0x7f665cc21100 [0x7f665cc20c80+0x480]
> J 14696 C2
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(Lorg/apache/hadoop/hbase/regionserver/HRegion;Lorg/apache/hadoop/hbase/quotas/OperationQuota;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionAction;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos$RegionActionResult$Builder;Ljava/util/List;JLorg/apache/hadoop/hbase/regionserver/RSRpcServices$RegionScannersCloseCallBack;Lorg/apache/hadoop/hbase/ipc/RpcCallContext;Lorg/apache/hadoop/hbase/quotas/ActivePolicyEnforcement;)Ljava/util/List;
> (901 bytes) @ 0x7f665b722148 [0x7f665b7218e0+0x868]
> {code}
> Here's another:
> {code}
> Stack: [0x7edd015e2000,0x7edd016e3000], sp=0x7edd016e11b0, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jbyte_disjoint_arraycopy
> J 18255 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyFromBufferToArray([BLjava/nio/ByteBuffer;III)V
> (69 bytes) @ 0x7f06d2593551 [0x7f06d2593460+0xf1]
> j
> org.apache.hadoop.hbase.PrivateCellUtil.copyTagsTo(Lorg/apache/hadoop/hbase/Cell;[BI)I+31
> j
> org.apache.hadoop.hbase.CellUtil.cloneTags(Lorg/apache/hadoop/hbase/Cell;)[B+12
> j org.apache.ha
[GitHub] [hbase] bsglz commented on pull request #1737: HBASE-24382 Flush partial stores of region filtered by seqId when arc…
bsglz commented on pull request #1737: URL: https://github.com/apache/hbase/pull/1737#issuecomment-649984842 Thanks all the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24382) Flush partial stores of region filtered by seqId when archive wal due to too many wals
[ https://issues.apache.org/jira/browse/HBASE-24382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146025#comment-17146025 ] Zheng Wang commented on HBASE-24382: Thanks a lot. > Flush partial stores of region filtered by seqId when archive wal due to too > many wals > -- > > Key: HBASE-24382 > URL: https://issues.apache.org/jira/browse/HBASE-24382 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When the logRoller archive the oldest wal due to too many wals, if a region > should be flushed, we flush all stores of it, but it is not necessary, maybe > we can use unflushedSeqId of store to filter them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase-native-client] bharathv commented on a change in pull request #6: HBASE-23105: Download lib double conversion, fizz, update folly
bharathv commented on a change in pull request #6:
URL: https://github.com/apache/hbase-native-client/pull/6#discussion_r445974129
##
File path: CMakeLists.txt
##
@@ -117,23 +124,22 @@ endif()
include_directories("${JAVA_HBASE_DIR}/hbase-common/target/generated-sources/native/")
-## Validate that we have C++ 14 support
+## Validate that we have C++ 17 support
Review comment:
Is this for std::optional?
##
File path: CMakeLists.txt
##
@@ -154,6 +160,9 @@ endif (OPENSSL_FOUND)
if (DOWNLOAD_DEPENDENCIES)
+ download_doubleconversion(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
+ download_fizz(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
Review comment:
This needs corresponding changes in Dockerfile?
##
File path: cmake/DownloadCyrusSasl.cmake
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download Cyrus SASL
+## SOURCE_DIR is typically the cmake source directory
+## BINARY_DIR is the build directory, typically 'build'
+
+
+function(download_cyrus_sasl SOURCE_DIR BUILD_DIR)
+ ExternalProject_Add(
+ cyrussasl
+ URL
"https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.27/cyrus-sasl-2.1.27.tar.gz";
+ PREFIX "${BUILD_DIR}/dependencies"
+ SOURCE_DIR "${BUILD_DIR}/dependencies/cyrussasl-src"
+ BINARY_DIR ${BUILD_DIR}/dependencies/cyrussasl-src/
+ CONFIGURE_COMMAND ./configure --enable-static --with-pic
--prefix=${BUILD_DIR}/dependencies/cyrussasl-install
+"CFLAGS=-fPIC"
+ "CXXFLAGS=${CMAKE_CXX_FLAGS} -fPIC"
+ )
+ add_library(sasl2 STATIC IMPORTED)
Review comment:
Move this to CMakeLists like other dependencies?
##
File path: cmake/DownloadWangle.cmake
##
@@ -28,16 +29,21 @@ function(download_wangle SOURCE_DIR BUILD_DIR)
else()
set(PATCH_FOLLY "")
endif()
-
+
ExternalProject_Add(
- facebook-wangle-proj
- URL "https://github.com/facebook/wangle/archive/v2017.09.04.00.tar.gz";
- PREFIX "${BUILD_DIR}/dependencies"
- DOWNLOAD_DIR ${WANGLE_DOWNLOAD_DIR}
- SOURCE_DIR ${WANGLE_SOURCE_DIR}
- PATCH_COMMAND ${PATCH_FOLLY}
- INSTALL_DIR ${WANGLE_INSTALL_DIR}
- CONFIGURE_COMMAND ${CMAKE_COMMAND} -DBUILD_EXAMPLES=OFF
-DCMAKE_CROSSCOMPILING=ON -DBUILD_TESTS=OFF -DFOLLY_ROOT_DIR=${FOLLY_ROOT_DIR}
-DCMAKE_INSTALL_PREFIX:PATH=${WANGLE_INSTALL_DIR} "${WANGLE_SOURCE_DIR}/wangle"
# Tell CMake to use subdirectory as source.
+ facebook-wangle-proj
+ GIT_REPOSITORY "https://github.com/facebook/wangle.git";
+ GIT_TAG "v2020.05.18.00"
+ SOURCE_DIR "${BUILD_DIR}/dependencies/facebook-wangle-proj-src"
Review comment:
Same comment as above, you've undone the PREFIX path, so just want to be
sure that the build directory is clean.
##
File path: cmake/DownloadDoubleConversion.cmake
##
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+function(download_doubleconversion SOURCE_DIR BUILD_DIR)
Review comment:
nit: 2-4 indentation (sorry for nit-picking :-))
##
File path: cmake/patches/zookeeper.3.4.14.buf
##
@@ -0,0 +1,4 @@
+3480c3480
+< static char buf[128];
Review comment:
unused?
##
File path: cmake/ProtobufGen.cmake
##
@@ -36,6 +36,7 @@ function(ge
[jira] [Created] (HBASE-24638) Edit doc on (offheap) memory management
Michael Stack created HBASE-24638: - Summary: Edit doc on (offheap) memory management Key: HBASE-24638 URL: https://issues.apache.org/jira/browse/HBASE-24638 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Michael Stack Gave it a read over to try and figure current state of memory management in hbase-2.3.0. Updated it to reflect more of what the current state is. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #1978: HBASE-24638 Edit doc on (offheap) memory management
saintstack opened a new pull request #1978: URL: https://github.com/apache/hbase/pull/1978 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1970: HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.
Apache-HBase commented on pull request #1970: URL: https://github.com/apache/hbase/pull/1970#issuecomment-650003425 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 58s | master passed | | +1 :green_heart: | compile | 1m 24s | master passed | | +1 :green_heart: | shadedjars | 5m 49s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-asyncfs in master failed. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | | -0 :warning: | patch | 7m 11s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 7s | the patch passed | | +1 :green_heart: | compile | 1m 23s | the patch passed | | +1 :green_heart: | javac | 1m 23s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-asyncfs in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 25s | hbase-asyncfs in the patch passed. | | +1 :green_heart: | unit | 128m 28s | hbase-server in the patch passed. | | | | 157m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1970 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b55e15b83043 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 84e246f9b1 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-asyncfs.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-asyncfs.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/testReport/ | | Max. process+thread count | 3870 (vs. ulimit of 12500) | | modules | C: hbase-asyncfs hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1970/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1978: HBASE-24638 Edit doc on (offheap) memory management
Apache-HBase commented on pull request #1978: URL: https://github.com/apache/hbase/pull/1978#issuecomment-650010126 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 41s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 6m 34s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | root in master failed. | | -0 :warning: | javadoc | 0m 17s | hbase-common in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 3m 5s | the patch passed | | +1 :green_heart: | javac | 3m 5s | the patch passed | | +1 :green_heart: | shadedjars | 6m 25s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 14s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 1m 18s | root in the patch failed. | | | | 33m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1978 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bbc62b9c84cb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / c0461207ee | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/testReport/ | | Max. process+thread count | 216 (vs. ulimit of 12500) | | modules | C: hbase-common . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1978/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
