[GitHub] [hbase] virajjasani commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
virajjasani commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573489127
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[jira] [Updated] (HBASE-25558) Adding audit log for execMasterService
[ https://issues.apache.org/jira/browse/HBASE-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lujie updated HBASE-25558: -- Issue Type: Improvement (was: Bug) > Adding audit log for execMasterService > -- > > Key: HBASE-25558 > URL: https://issues.apache.org/jira/browse/HBASE-25558 > Project: HBase > Issue Type: Improvement >Reporter: lujie >Priority: Major > > I think we should add audit log for execMasterService, like execProcedure -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] arshadmohammad commented on pull request #2883: HBASE-25492: Create table with rsgroup
arshadmohammad commented on pull request #2883: URL: https://github.com/apache/hbase/pull/2883#issuecomment-776441988 @saintstack please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2946: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2.4
Apache-HBase commented on pull request #2946: URL: https://github.com/apache/hbase/pull/2946#issuecomment-776417872 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 16s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 47s | branch-2.4 passed | | +1 :green_heart: | compile | 0m 59s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 1s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 5m 56s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 148m 7s | hbase-server in the patch passed. | | | | 173m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2946 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c91738da9d24 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / e7414f98fd | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/testReport/ | | Max. process+thread count | 3797 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2946: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2.4
Apache-HBase commented on pull request #2946: URL: https://github.com/apache/hbase/pull/2946#issuecomment-776414907 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 16s | branch-2.4 passed | | +1 :green_heart: | compile | 1m 5s | branch-2.4 passed | | +1 :green_heart: | shadedjars | 6m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 132m 29s | hbase-server in the patch passed. | | | | 163m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2946 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 966563a87282 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / e7414f98fd | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/testReport/ | | Max. process+thread count | 3834 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282171#comment-17282171
]
Hudson commented on HBASE-25507:
Results for branch branch-2.4
[build #48 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.5, 2.4.2
>
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.had
[jira] [Commented] (HBASE-25559) Terminate threads of oldsources while RS is closing
[
https://issues.apache.org/jira/browse/HBASE-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282172#comment-17282172
]
Hudson commented on HBASE-25559:
Results for branch branch-2.4
[build #48 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Terminate threads of oldsources while RS is closing
> ---
>
> Key: HBASE-25559
> URL: https://issues.apache.org/jira/browse/HBASE-25559
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.3.4, 2.4.1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25542) Add client detail to scan name so when lease expires, we have clue on who was scanning
[
https://issues.apache.org/jira/browse/HBASE-25542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282170#comment-17282170
]
Hudson commented on HBASE-25542:
Results for branch branch-2.4
[build #48 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add client detail to scan name so when lease expires, we have clue on who was
> scanning
> --
>
> Key: HBASE-25542
> URL: https://issues.apache.org/jira/browse/HBASE-25542
> Project: HBase
> Issue Type: Improvement
> Components: scan
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2
>
>
> On a busy cluster, seeing a bunch of this:
> {code}
> regionserver.RSRpcServices: Scanner -5594731657924732088 lease expired on
> region hbase:meta,,1.1588230740
> {code}
> Trying to figure out originator is a little tough; there isn't enough in for
> in the above.
> scanid is a long but scan name is a string.. perhaps add in the client
> ip+port as scan id prefix? Or add the info to the value -- probably better --
> and dump it out when we get one of these expirations?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25431) MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number
[
https://issues.apache.org/jira/browse/HBASE-25431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282169#comment-17282169
]
Hudson commented on HBASE-25431:
Results for branch branch-2.4
[build #48 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.4/48/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number
> -
>
> Key: HBASE-25431
> URL: https://issues.apache.org/jira/browse/HBASE-25431
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: image-2020-12-22-11-46-38-967.png
>
>
> Before create or alter table, we will do sanityCheck in
> TableDescriptorChecker. If MAX_FILESIZE or MEMSTORE_FLUSHSIZE < 0, use
> maxFileSizeLowerLimit or flushSizeLowerLimit instead to pass check. But the
> real value in TableDescriptor is still < 0, and we can see negative values
> on the UI.
> However in flush and split logic, MAX_FILESIZE and MEMSTORE_FLUSHSIZE will
> judge whether it's value <= 0 , if true, change to default value. This does
> not affect flush and split.
> !image-2020-12-22-11-46-38-967.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2946: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2.4
Apache-HBase commented on pull request #2946: URL: https://github.com/apache/hbase/pull/2946#issuecomment-776368521 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2.4 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 4s | branch-2.4 passed | | +1 :green_heart: | compile | 3m 22s | branch-2.4 passed | | +1 :green_heart: | checkstyle | 1m 17s | branch-2.4 passed | | +1 :green_heart: | spotbugs | 2m 11s | branch-2.4 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 38s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | +1 :green_heart: | javac | 3m 21s | the patch passed | | +1 :green_heart: | checkstyle | 1m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 1s | Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 49m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2946 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 57b8e57df286 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2.4 / e7414f98fd | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 86 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2946/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25534:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master and branch-2.4. Thanks for the contribution [~DeanZ] !
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.2
>
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #2946: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2.4
ndimiduk merged pull request #2946: URL: https://github.com/apache/hbase/pull/2946 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25534:
-
Fix Version/s: 2.4.2
3.0.0-alpha-1
Status: Patch Available (was: Open)
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.2
>
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2945: [HBASE-25539] Adding replication log queue
Apache-HBase commented on pull request #2945: URL: https://github.com/apache/hbase/pull/2945#issuecomment-776350616 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 11s | master passed | | +1 :green_heart: | shadedjars | 7m 28s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 44s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 1m 11s | the patch passed | | +1 :green_heart: | javac | 1m 11s | the patch passed | | -1 :x: | shadedjars | 5m 57s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 199m 12s | hbase-server in the patch passed. | | | | 228m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2945 | | JIRA Issue | HBASE-25539 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c914e18960fb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-11.0.6+10 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/testReport/ | | Max. process+thread count | 3096 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #2946: Backport "HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)" to branch-2.4
ndimiduk opened a new pull request #2946: URL: https://github.com/apache/hbase/pull/2946 Signed-off-by: Nick Dimiduk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25534) Honor TableDescriptor settings earlier in normalization
[
https://issues.apache.org/jira/browse/HBASE-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-25534:
-
Summary: Honor TableDescriptor settings earlier in normalization (was:
Improve the configuration of Normalizer)
> Honor TableDescriptor settings earlier in normalization
> ---
>
> Key: HBASE-25534
> URL: https://issues.apache.org/jira/browse/HBASE-25534
> Project: HBase
> Issue Type: Improvement
> Components: Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
>
> -Table can be enabled and disabled merge/split in TableDescriptor, we should
> judge before calculating the plan.-
> The normalizer configuration can be set by table level. For example,
> hbase.normalizer.min.region.count can be set by "alter ‘table’,
> CONFIGURATION=>\{'hbase.normalizer.min.region.count' => '5'}". If the table
> is not set, then use the global configuration which is set in hbase-site.xml.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #2917: HBASE-25534 Honor TableDescriptor settings earlier in normalization
ndimiduk merged pull request #2917: URL: https://github.com/apache/hbase/pull/2917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2945: [HBASE-25539] Adding replication log queue
Apache-HBase commented on pull request #2945: URL: https://github.com/apache/hbase/pull/2945#issuecomment-776325230 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 6m 32s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | -1 :x: | shadedjars | 5m 15s | patch has 10 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 146m 42s | hbase-server in the patch passed. | | | | 171m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2945 | | JIRA Issue | HBASE-25539 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 3c6219280044 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/testReport/ | | Max. process+thread count | 4190 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2934: HBASE-23887 AdaptiveLRU cache
Apache-HBase commented on pull request #2934: URL: https://github.com/apache/hbase/pull/2934#issuecomment-776303080 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 6m 44s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 30s | the patch passed | | +1 :green_heart: | compile | 1m 0s | the patch passed | | +1 :green_heart: | javac | 1m 0s | the patch passed | | +1 :green_heart: | shadedjars | 6m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 27s | hbase-server in the patch passed. | | | | 168m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2934 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 815a82d13f77 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/testReport/ | | Max. process+thread count | 4259 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2934: HBASE-23887 AdaptiveLRU cache
Apache-HBase commented on pull request #2934: URL: https://github.com/apache/hbase/pull/2934#issuecomment-776300810 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 22s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 6m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 135m 38s | hbase-server in the patch passed. | | | | 163m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2934 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d62f296764ff 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/testReport/ | | Max. process+thread count | 3964 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] shahrs87 commented on a change in pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying
shahrs87 commented on a change in pull request #2943:
URL: https://github.com/apache/hbase/pull/2943#discussion_r573298803
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java
##
@@ -148,12 +148,10 @@ public void run() {
}
}
} catch (IOException e) { // stream related
+LOG.error("Failed to read stream of replication entries", e);
Review comment:
@wchevreuil We will log inside handleEofException *only* if the
exception or cause of exception is an EOF not otherwise. If we make this line
at debug then we won't see any logs if it is caused by non-eof exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2945: [HBASE-25539] Adding replication log queue
Apache-HBase commented on pull request #2945: URL: https://github.com/apache/hbase/pull/2945#issuecomment-776265180 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 25s | master passed | | +1 :green_heart: | compile | 3m 8s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 6 new + 3 unchanged - 1 fixed = 9 total (was 4) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 16m 58s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | -1 :x: | spotbugs | 2m 14s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 13s | The patch generated 1 ASF License warnings. | | | | 44m 33s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Dead store to queue in org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.enqueueLog(Path) At ReplicationSource.java:org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.enqueueLog(Path) At ReplicationSource.java:[line 254] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2945 | | JIRA Issue | HBASE-25539 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 692e59a29435 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2934: HBASE-23887 AdaptiveLRU cache
Apache-HBase commented on pull request #2934: URL: https://github.com/apache/hbase/pull/2934#issuecomment-776241222 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | compile | 3m 10s | master passed | | +1 :green_heart: | checkstyle | 1m 3s | master passed | | +1 :green_heart: | spotbugs | 2m 1s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 3m 6s | the patch passed | | +1 :green_heart: | javac | 3m 6s | the patch passed | | -0 :warning: | checkstyle | 1m 5s | hbase-server: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 17m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 45m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2934 | | Optional Tests | dupname asflicense spotbugs xml javac hadoopcheck hbaseanti checkstyle compile | | uname | Linux 5f6590ced09f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2934/6/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2945: [HBASE-25539] Adding replication log queue
Apache-HBase commented on pull request #2945: URL: https://github.com/apache/hbase/pull/2945#issuecomment-776235323 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2945: [HBASE-25539] Adding replication log queue
Apache-HBase commented on pull request #2945: URL: https://github.com/apache/hbase/pull/2945#issuecomment-776231613 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | ||| _ Patch Compile Tests _ | | -1 :x: | mvninstall | 2m 6s | root in the patch failed. | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | -1 :x: | shadedjars | 5m 16s | patch has 11 errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 0m 58s | hbase-server in the patch failed. | | | | 23m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2945 | | JIRA Issue | HBASE-25539 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 19c0dac5b3e4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | mvninstall | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt | | shadedjars | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/testReport/ | | Max. process+thread count | 88 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2945/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282015#comment-17282015
]
Danil Lipovoy commented on HBASE-23887:
---
[~vjasani] Thanks for reasonable comments! Fixed
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] shahrs87 opened a new pull request #2945: [HBASE-25539] Adding replication log queue
shahrs87 opened a new pull request #2945: URL: https://github.com/apache/hbase/pull/2945 Still in draft phase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573208568
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573208486
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573208223
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573208070
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573207460
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573207051
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573206391
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573206341
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573206052
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[GitHub] [hbase] pustota2009 commented on a change in pull request #2934: HBASE-23887 AdaptiveLRU cache
pustota2009 commented on a change in pull request #2934:
URL: https://github.com/apache/hbase/pull/2934#discussion_r573205983
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruAdaptiveBlockCache.java
##
@@ -0,0 +1,1442 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.io.hfile;
+
+import static java.util.Objects.requireNonNull;
+
+import java.lang.ref.WeakReference;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.PriorityQueue;
+import java.util.SortedSet;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.io.HeapSize;
+import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
+import org.apache.hadoop.hbase.util.ClassSize;
+import org.apache.hadoop.util.StringUtils;
+import org.apache.yetus.audience.InterfaceAudience;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hbase.thirdparty.com.google.common.base.MoreObjects;
+import org.apache.hbase.thirdparty.com.google.common.base.Objects;
+import
org.apache.hbase.thirdparty.com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+/**
+ * This realisation improve performance of classical LRU
+ * cache up to 3 times via reduce GC job.
+ *
+ * The classical block cache implementation that is memory-aware using {@link
HeapSize},
+ * memory-bound using an
+ * LRU eviction algorithm, and concurrent: backed by a {@link
ConcurrentHashMap} and with a
+ * non-blocking eviction thread giving constant-time {@link #cacheBlock} and
{@link #getBlock}
+ * operations.
+ *
+ * Contains three levels of block priority to allow for scan-resistance and
in-memory families
+ * {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder#setInMemory(boolean)}
(An
+ * in-memory column family is a column family that should be served from
memory if possible):
+ * single-access, multiple-accesses, and in-memory priority. A block is added
with an in-memory
+ * priority flag if {@link
org.apache.hadoop.hbase.client.ColumnFamilyDescriptor#isInMemory()},
+ * otherwise a block becomes a single access priority the first time it is
read into this block
+ * cache. If a block is accessed again while in cache, it is marked as a
multiple access priority
+ * block. This delineation of blocks is used to prevent scans from thrashing
the cache adding a
+ * least-frequently-used element to the eviction algorithm.
+ *
+ * Each priority is given its own chunk of the total cache to ensure fairness
during eviction. Each
+ * priority will retain close to its maximum size, however, if any priority is
not using its entire
+ * chunk the others are able to grow beyond their chunk size.
+ *
+ * Instantiated at a minimum with the total size and average block size. All
sizes are in bytes. The
+ * block size is not especially important as this cache is fully dynamic in
its sizing of blocks. It
+ * is only used for pre-allocating data structures and in initial heap
estimation of the map.
+ *
+ * The detailed constructor defines the sizes for the three priorities (they
should total to the
+ * maximum size defined). It also sets the levels that trigger
and control the eviction
+ * thread.
+ *
+ * The acceptable size is the cache size level which triggers the
eviction process to
+ * start. It evicts enough blocks to get the size below the minimum size
specified.
+ *
+ * Eviction happens in a separate thread and involves a single full-scan of
the map. It determines
+ * how many bytes must be freed to reach the minimum size, and then while
scanning determines the
+ * fewest least-recently-used blocks necessary from each of the three
priorities (would be 3 times
+ * bytes to free). It then uses the priority chunk sizes to evict fairly
according to the relative
[jira] [Commented] (HBASE-25559) Terminate threads of oldsources while RS is closing
[
https://issues.apache.org/jira/browse/HBASE-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281976#comment-17281976
]
Hudson commented on HBASE-25559:
Results for branch branch-2.2
[build #168 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/168/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/168//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/168//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/168//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.2/168//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Terminate threads of oldsources while RS is closing
> ---
>
> Key: HBASE-25559
> URL: https://issues.apache.org/jira/browse/HBASE-25559
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.3.4, 2.4.1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25512) May throw StringIndexOutOfBoundsException when construct illegal tablename error
[
https://issues.apache.org/jira/browse/HBASE-25512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281946#comment-17281946
]
Hudson commented on HBASE-25512:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> May throw StringIndexOutOfBoundsException when construct illegal tablename
> error
>
>
> Key: HBASE-25512
> URL: https://issues.apache.org/jira/browse/HBASE-25512
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.12, 1.4.13, 2.4.1
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When call Method:
> {code:java}
> // code placeholder
> TableName.isLegalTableQualifierName(final byte[] qualifierName, int start,
> int end)
> {code}
> We want to construct quelifier String to print pretty error message. So we
> call method:
> {code:java}
> // code placeholder
> Bytes.toString(final byte[] b, int off, int len)
> Bytes.toString(qualifierName, start, end)
> {code}
> But the param is wrong, we shoud pass *${Length}* instead of *${end index}*
> to *Bytes.toString third param*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281948#comment-17281948
]
Hudson commented on HBASE-25507:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.5, 2.4.2
>
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apache.hadoop.hb
[jira] [Commented] (HBASE-25550) More readable Competition Time
[
https://issues.apache.org/jira/browse/HBASE-25550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281944#comment-17281944
]
Hudson commented on HBASE-25550:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> More readable Competition Time
> --
>
> Key: HBASE-25550
> URL: https://issues.apache.org/jira/browse/HBASE-25550
> Project: HBase
> Issue Type: Sub-task
> Components: UI
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: image-2021-02-04-15-21-00-019.png
>
>
> Should add duration to Completion Time. Make it more readable
> !https://internal-api-lark-file.f.mioffice.cn/api/image/keys/img_44a98ccd-4576-4971-900e-3d27dd1263el?message_id=6925292609535492204!
> to
> !image-2021-02-04-15-21-00-019.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25560) Remove unused parameter named peerId in the constructor method of CatalogReplicationSourcePeer
[
https://issues.apache.org/jira/browse/HBASE-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281943#comment-17281943
]
Hudson commented on HBASE-25560:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove unused parameter named peerId in the constructor method of
> CatalogReplicationSourcePeer
> --
>
> Key: HBASE-25560
> URL: https://issues.apache.org/jira/browse/HBASE-25560
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25559) Terminate threads of oldsources while RS is closing
[
https://issues.apache.org/jira/browse/HBASE-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281949#comment-17281949
]
Hudson commented on HBASE-25559:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Terminate threads of oldsources while RS is closing
> ---
>
> Key: HBASE-25559
> URL: https://issues.apache.org/jira/browse/HBASE-25559
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.3.4, 2.4.1
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24772) Use GetoptLong or OptionParser in hbase-shell
[
https://issues.apache.org/jira/browse/HBASE-24772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281947#comment-17281947
]
Hudson commented on HBASE-24772:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Use GetoptLong or OptionParser in hbase-shell
> -
>
> Key: HBASE-24772
> URL: https://issues.apache.org/jira/browse/HBASE-24772
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Elliot Miller
>Assignee: Laxman Goswami
>Priority: Minor
> Labels: beginner-friendly
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Currently, our hbase-shell command line argument parser is custom-rolled. It
> would be awesome to instead use Ruby's GetoptLong or OptionParser.
> * [https://ruby-doc.org/stdlib-2.3.0/libdoc/getoptlong/rdoc/GetoptLong.html]
> * [https://ruby-doc.org/stdlib-2.3.0/libdoc/optparse/rdoc/OptionParser.html]
> There is a long-standing {{FIXME}} comment in {{bin/hirb.rb}} to address
> this:
> [https://github.com/apache/hbase/blob/975cdf7b88f001aa51d83395c663892f3b5ffbdb/bin/hirb.rb#L50]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25542) Add client detail to scan name so when lease expires, we have clue on who was scanning
[
https://issues.apache.org/jira/browse/HBASE-25542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281945#comment-17281945
]
Hudson commented on HBASE-25542:
Results for branch branch-2
[build #173 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/173/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add client detail to scan name so when lease expires, we have clue on who was
> scanning
> --
>
> Key: HBASE-25542
> URL: https://issues.apache.org/jira/browse/HBASE-25542
> Project: HBase
> Issue Type: Improvement
> Components: scan
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2
>
>
> On a busy cluster, seeing a bunch of this:
> {code}
> regionserver.RSRpcServices: Scanner -5594731657924732088 lease expired on
> region hbase:meta,,1.1588230740
> {code}
> Trying to figure out originator is a little tough; there isn't enough in for
> in the above.
> scanid is a long but scan name is a string.. perhaps add in the client
> ip+port as scan id prefix? Or add the info to the value -- probably better --
> and dump it out when we get one of these expirations?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25507) Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic"
[
https://issues.apache.org/jira/browse/HBASE-25507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281903#comment-17281903
]
Hudson commented on HBASE-25507:
Results for branch branch-2.3
[build #166 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/166/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/166/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/166/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/166/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2.3/166/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException:
> Invalid HFile block magic"
>
>
> Key: HBASE-25507
> URL: https://issues.apache.org/jira/browse/HBASE-25507
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Affects Versions: 3.0.0-alpha-1, 2.4.1
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.5, 2.4.2
>
> Attachments: errorlogs.png,
> increasing-of-established-sockets-image.png, problem-region-move-logs.png
>
>
> Recently, we found socket leaks on our production cluster. The leaked sockets
> are in ESTABLISHED state. We found this happened on RS who owned a particular
> region from our analysis of metrics monitor and logs. RS without this region
> works normally.
> On the RS who owns the particular region, we found Exceptions as follows,
> {code:java}
> java.io.IOException: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.ioo
> .HalfStoreFileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFF
> amilyCell@25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.parallelSeek(StoreScanner.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:437)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:302)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:88
> 06)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor$StripeInternalScannerFacc
> tory.createScanner(StripeCompactor.java:82)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:316)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactor.compact(StripeCompactor..
> java:120)
> at
> org.apache.hadoop.hbase.regionserver.compactions.StripeCompactionPolicy$SplitStripeCompacc
> tionRequest.execute(StripeCompactionPolicy.java:662)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreEngine$StripeCompaction.compact(StripeStoo
> reEngine.java:114)
> at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1461)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2121)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(Comm
> pactSplitThread.java:519)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitt
> Thread.java:555)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Could not seek
> StoreFileScanner[org.apache.hadoop.hbase.io.HalfStoree
> FileReader$1@5388be2f, cur=null] to key
> org.apache.hadoop.hbase.CellUtil$FirstOnRowDeleteFamilyCell@@
> 25aa56fd
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:229)
> at
> org.apac
[GitHub] [hbase] Apache-HBase commented on pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
Apache-HBase commented on pull request #2780: URL: https://github.com/apache/hbase/pull/2780#issuecomment-776074292 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 7s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 40s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 42s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 146m 39s | hbase-server in the patch passed. | | | | 173m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2780 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 969de7179e8e 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/testReport/ | | Max. process+thread count | 4313 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying
saintstack commented on a change in pull request #2943:
URL: https://github.com/apache/hbase/pull/2943#discussion_r573040113
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java
##
@@ -148,12 +148,10 @@ public void run() {
}
}
} catch (IOException e) { // stream related
+LOG.error("Failed to read stream of replication entries", e);
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
Apache-HBase commented on pull request #2780: URL: https://github.com/apache/hbase/pull/2780#issuecomment-776070593 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 41s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 1m 7s | the patch passed | | +1 :green_heart: | javac | 1m 7s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 139m 53s | hbase-server in the patch passed. | | | | 168m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2780 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5537c4540b82 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/testReport/ | | Max. process+thread count | 3843 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2944: HBASE-25519 BLOCKSIZE needs to support pretty print (branch-2)
Apache-HBase commented on pull request #2944: URL: https://github.com/apache/hbase/pull/2944#issuecomment-776041004 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | branch-2 passed | | +1 :green_heart: | compile | 1m 32s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 45s | branch-2 passed | | +1 :green_heart: | spotbugs | 1m 11s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 40s | the patch passed | | +1 :green_heart: | compile | 1m 27s | the patch passed | | +1 :green_heart: | javac | 1m 27s | the patch passed | | +1 :green_heart: | checkstyle | 0m 45s | the patch passed | | -0 :warning: | rubocop | 0m 7s | The patch generated 3 new + 361 unchanged - 3 fixed = 364 total (was 364) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 2s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 22s | The patch does not generate ASF License warnings. | | | | 38m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2944 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile rubocop | | uname | Linux b2cae9151fd1 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b05dcac9fd | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | rubocop | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | Max. process+thread count | 86 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2944: HBASE-25519 BLOCKSIZE needs to support pretty print (branch-2)
Apache-HBase commented on pull request #2944: URL: https://github.com/apache/hbase/pull/2944#issuecomment-776039914 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | branch-2 passed | | +1 :green_heart: | compile | 0m 53s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 48s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 45s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 0m 53s | the patch passed | | +1 :green_heart: | javac | 0m 53s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 25s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 6m 53s | hbase-shell in the patch passed. | | | | 36m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2944 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 5568a2d52622 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b05dcac9fd | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/testReport/ | | Max. process+thread count | 2039 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2944: HBASE-25519 BLOCKSIZE needs to support pretty print (branch-2)
Apache-HBase commented on pull request #2944: URL: https://github.com/apache/hbase/pull/2944#issuecomment-776036652 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 36s | branch-2 passed | | +1 :green_heart: | compile | 0m 51s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 2s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 18s | the patch passed | | +1 :green_heart: | compile | 0m 51s | the patch passed | | +1 :green_heart: | javac | 0m 51s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 24s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 6m 58s | hbase-shell in the patch passed. | | | | 33m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2944 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df76e8c7dda2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / b05dcac9fd | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/testReport/ | | Max. process+thread count | 1952 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-shell U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2944/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25519) BLOCKSIZE needs to support pretty print
[ https://issues.apache.org/jira/browse/HBASE-25519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25519. --- Fix Version/s: 2.5.0 Resolution: Fixed Resovling again after applying to branch-2. Thanks for the backport [~DeanZ]. > BLOCKSIZE needs to support pretty print > --- > > Key: HBASE-25519 > URL: https://issues.apache.org/jira/browse/HBASE-25519 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > HBASE-25439 added a new unit in PrettyPrint.Unit, the BLOCKSIZE in CF should > also support this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-25519) BLOCKSIZE needs to support pretty print
[ https://issues.apache.org/jira/browse/HBASE-25519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-25519: --- Reopen to apply branch-2 PR > BLOCKSIZE needs to support pretty print > --- > > Key: HBASE-25519 > URL: https://issues.apache.org/jira/browse/HBASE-25519 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1 > > > HBASE-25439 added a new unit in PrettyPrint.Unit, the BLOCKSIZE in CF should > also support this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2944: HBASE-25519 BLOCKSIZE needs to support pretty print (branch-2)
saintstack merged pull request #2944: URL: https://github.com/apache/hbase/pull/2944 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25519) BLOCKSIZE needs to support pretty print
[ https://issues.apache.org/jira/browse/HBASE-25519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281833#comment-17281833 ] Baiqiang Zhao commented on HBASE-25519: --- Thanks for review [~stack]. PR for branch-2 has been put up. > BLOCKSIZE needs to support pretty print > --- > > Key: HBASE-25519 > URL: https://issues.apache.org/jira/browse/HBASE-25519 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.5.0 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1 > > > HBASE-25439 added a new unit in PrettyPrint.Unit, the BLOCKSIZE in CF should > also support this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ZhaoBQ opened a new pull request #2944: HBASE-25519 BLOCKSIZE needs to support pretty print (branch-2)
ZhaoBQ opened a new pull request #2944: URL: https://github.com/apache/hbase/pull/2944 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
Apache-HBase commented on pull request #2780: URL: https://github.com/apache/hbase/pull/2780#issuecomment-775983543 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | compile | 3m 2s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 2s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 27s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 17m 11s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 45m 12s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2780 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux d4ad3b65c775 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 4a3ff98943 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2780/2/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281808#comment-17281808
]
Viraj Jasani commented on HBASE-23887:
--
[~pustota], left few comments on PR. Please take a look when you get time.
Thanks
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, PR#1257.diff,
> cmp.png, evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, image-2020-09-23-10-06-11-189.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png, ycsb_logs.zip
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] sanjeetnishad95 commented on a change in pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
sanjeetnishad95 commented on a change in pull request #2780:
URL: https://github.com/apache/hbase/pull/2780#discussion_r572895744
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
##
@@ -621,4 +622,38 @@ public void testTableConstraint() throws Exception {
TEST_UTIL.deleteTable(tableName);
ADMIN.deleteNamespace(ns);
}
+
+ @Test
+ public void testHBASE25301() throws IOException, InterruptedException {
+String pgroup = "pgroup";
+ADMIN.addRSGroup(pgroup);
+
+ServerName serverName0 =
TEST_UTIL.getMiniHBaseCluster().getRegionServer(0).getServerName();
+ServerName serverName1 =
TEST_UTIL.getMiniHBaseCluster().getRegionServer(1).getServerName();
+ADMIN.moveServersToRSGroup(Sets.newHashSet(serverName0.getAddress(),
serverName1.getAddress()),
+ pgroup);
+
+TableName table1 = TableName.valueOf("table1");
+TableName table2 = TableName.valueOf("table2");
+TEST_UTIL.createTable(table1, "cf");
+TEST_UTIL.createTable(table2, "cf");
+
+ADMIN.setRSGroup(Sets.newHashSet(table1, table2), pgroup);
+
+List regionInfoList = ADMIN.getRegions(table1);
+regionInfoList.addAll(ADMIN.getRegions(table2));
+for (RegionInfo regionInfo : regionInfoList) {
+ ADMIN.move(regionInfo.getEncodedNameAsBytes(), serverName0);
+}
+
+ADMIN.split(table2, "50".getBytes());
+// Waiting briefly so that daughter regions are created but parent is not
cleaned yet.
+Thread.sleep(2000);
Review comment:
Added a wait logic to wait enough for parent region to have state as
`SPLIT`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sanjeetnishad95 commented on a change in pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
sanjeetnishad95 commented on a change in pull request #2780:
URL: https://github.com/apache/hbase/pull/2780#discussion_r572895333
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
##
@@ -621,4 +622,38 @@ public void testTableConstraint() throws Exception {
TEST_UTIL.deleteTable(tableName);
ADMIN.deleteNamespace(ns);
}
+
+ @Test
+ public void testHBASE25301() throws IOException, InterruptedException {
+String pgroup = "pgroup";
+ADMIN.addRSGroup(pgroup);
+
+ServerName serverName0 =
TEST_UTIL.getMiniHBaseCluster().getRegionServer(0).getServerName();
+ServerName serverName1 =
TEST_UTIL.getMiniHBaseCluster().getRegionServer(1).getServerName();
+ADMIN.moveServersToRSGroup(Sets.newHashSet(serverName0.getAddress(),
serverName1.getAddress()),
+ pgroup);
+
+TableName table1 = TableName.valueOf("table1");
+TableName table2 = TableName.valueOf("table2");
+TEST_UTIL.createTable(table1, "cf");
+TEST_UTIL.createTable(table2, "cf");
+
+ADMIN.setRSGroup(Sets.newHashSet(table1, table2), pgroup);
+
+List regionInfoList = ADMIN.getRegions(table1);
+regionInfoList.addAll(ADMIN.getRegions(table2));
+for (RegionInfo regionInfo : regionInfoList) {
Review comment:
Added a check to move if region is not already in the target server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sanjeetnishad95 commented on a change in pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
sanjeetnishad95 commented on a change in pull request #2780:
URL: https://github.com/apache/hbase/pull/2780#discussion_r572894873
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin1.java
##
@@ -621,4 +622,38 @@ public void testTableConstraint() throws Exception {
TEST_UTIL.deleteTable(tableName);
ADMIN.deleteNamespace(ns);
}
+
+ @Test
+ public void testHBASE25301() throws IOException, InterruptedException {
Review comment:
I have changed the method name and updated the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] sanjeetnishad95 commented on a change in pull request #2780: HBASE-25301 NPE while running balance_rsgroup if any split region is …
sanjeetnishad95 commented on a change in pull request #2780:
URL: https://github.com/apache/hbase/pull/2780#discussion_r572894248
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java
##
@@ -1130,6 +1130,12 @@ private boolean isTableInGroup(TableName tableName,
String groupName,
if (region.isSplitParent()) {
continue;
}
+
+// isSplitParent() and isSplit() is not always reliable. So for those
scenarios, better to
Review comment:
Hi Pankaj, thanks for reviewing. Found a pending TODO in
`org.apache.hadoop.hbase.master.assignment.AssignmentManager#markRegionAsSplit`
which explains why isSplit() is not reliable.
I have updated the MR by checking the region state instead of null server.
I am also planning to open another JIRA for resolving that TODO if come up
with a solution.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-25512) May throw StringIndexOutOfBoundsException when construct illegal tablename error
[
https://issues.apache.org/jira/browse/HBASE-25512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281777#comment-17281777
]
Hudson commented on HBASE-25512:
Results for branch master
[build #206 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> May throw StringIndexOutOfBoundsException when construct illegal tablename
> error
>
>
> Key: HBASE-25512
> URL: https://issues.apache.org/jira/browse/HBASE-25512
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.12, 1.4.13, 2.4.1
>Reporter: zhuobin zheng
>Assignee: zhuobin zheng
>Priority: Trivial
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When call Method:
> {code:java}
> // code placeholder
> TableName.isLegalTableQualifierName(final byte[] qualifierName, int start,
> int end)
> {code}
> We want to construct quelifier String to print pretty error message. So we
> call method:
> {code:java}
> // code placeholder
> Bytes.toString(final byte[] b, int off, int len)
> Bytes.toString(qualifierName, start, end)
> {code}
> But the param is wrong, we shoud pass *${Length}* instead of *${end index}*
> to *Bytes.toString third param*.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25542) Add client detail to scan name so when lease expires, we have clue on who was scanning
[
https://issues.apache.org/jira/browse/HBASE-25542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281778#comment-17281778
]
Hudson commented on HBASE-25542:
Results for branch master
[build #206 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add client detail to scan name so when lease expires, we have clue on who was
> scanning
> --
>
> Key: HBASE-25542
> URL: https://issues.apache.org/jira/browse/HBASE-25542
> Project: HBase
> Issue Type: Improvement
> Components: scan
>Affects Versions: 2.3.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.2
>
>
> On a busy cluster, seeing a bunch of this:
> {code}
> regionserver.RSRpcServices: Scanner -5594731657924732088 lease expired on
> region hbase:meta,,1.1588230740
> {code}
> Trying to figure out originator is a little tough; there isn't enough in for
> in the above.
> scanid is a long but scan name is a string.. perhaps add in the client
> ip+port as scan id prefix? Or add the info to the value -- probably better --
> and dump it out when we get one of these expirations?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25550) More readable Competition Time
[
https://issues.apache.org/jira/browse/HBASE-25550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281775#comment-17281775
]
Hudson commented on HBASE-25550:
Results for branch master
[build #206 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> More readable Competition Time
> --
>
> Key: HBASE-25550
> URL: https://issues.apache.org/jira/browse/HBASE-25550
> Project: HBase
> Issue Type: Sub-task
> Components: UI
>Reporter: Zhuoyue Huang
>Assignee: Zhuoyue Huang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: image-2021-02-04-15-21-00-019.png
>
>
> Should add duration to Completion Time. Make it more readable
> !https://internal-api-lark-file.f.mioffice.cn/api/image/keys/img_44a98ccd-4576-4971-900e-3d27dd1263el?message_id=6925292609535492204!
> to
> !image-2021-02-04-15-21-00-019.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25519) BLOCKSIZE needs to support pretty print
[
https://issues.apache.org/jira/browse/HBASE-25519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281776#comment-17281776
]
Hudson commented on HBASE-25519:
Results for branch master
[build #206 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> BLOCKSIZE needs to support pretty print
> ---
>
> Key: HBASE-25519
> URL: https://issues.apache.org/jira/browse/HBASE-25519
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> HBASE-25439 added a new unit in PrettyPrint.Unit, the BLOCKSIZE in CF should
> also support this feature.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25560) Remove unused parameter named peerId in the constructor method of CatalogReplicationSourcePeer
[
https://issues.apache.org/jira/browse/HBASE-25560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281774#comment-17281774
]
Hudson commented on HBASE-25560:
Results for branch master
[build #206 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/206/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove unused parameter named peerId in the constructor method of
> CatalogReplicationSourcePeer
> --
>
> Key: HBASE-25560
> URL: https://issues.apache.org/jira/browse/HBASE-25560
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.4.2
>Reporter: Sun Xin
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying
Apache-HBase commented on pull request #2943: URL: https://github.com/apache/hbase/pull/2943#issuecomment-775896915 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 59s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 10s | master passed | | +1 :green_heart: | shadedjars | 7m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 43s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 7m 22s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 200m 50s | hbase-server in the patch passed. | | | | 232m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2943 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1bd5c68fed36 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0353909bc2 | | Default Java | AdoptOpenJDK-11.0.6+10 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/testReport/ | | Max. process+thread count | 2871 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying
Apache-HBase commented on pull request #2943: URL: https://github.com/apache/hbase/pull/2943#issuecomment-775865394 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 25s | master passed | | +1 :green_heart: | compile | 1m 7s | master passed | | +1 :green_heart: | shadedjars | 7m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 53s | the patch passed | | +1 :green_heart: | compile | 0m 57s | the patch passed | | +1 :green_heart: | javac | 0m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 37s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 20s | hbase-server in the patch passed. | | | | 170m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2943 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 76f327842f9d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0353909bc2 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/testReport/ | | Max. process+thread count | 4316 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying
wchevreuil commented on a change in pull request #2943:
URL: https://github.com/apache/hbase/pull/2943#discussion_r572766045
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceWALReader.java
##
@@ -148,12 +148,10 @@ public void run() {
}
}
} catch (IOException e) { // stream related
+LOG.error("Failed to read stream of replication entries", e);
Review comment:
We already log about the EOF issue inside _handleEofException_, so maybe
worth keep this one as debug to avoid flooding the log while we keep retrying?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (HBASE-25559) Terminate threads of oldsources while RS is closing
[ https://issues.apache.org/jira/browse/HBASE-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281654#comment-17281654 ] Sun Xin edited comment on HBASE-25559 at 2/9/21, 9:29 AM: -- Merged to master and all active branch-2.x. Thanks [~wchevreuil] [~vjasani] [~stack] for reviwing. was (Author: ddupg): Merged to master and all active branch-2.x. > Terminate threads of oldsources while RS is closing > --- > > Key: HBASE-25559 > URL: https://issues.apache.org/jira/browse/HBASE-25559 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.3.4, 2.4.1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] aimiebell commented on a change in pull request #2936: HBASE-21674 Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
aimiebell commented on a change in pull request #2936: URL: https://github.com/apache/hbase/pull/2936#discussion_r572623116 ## File path: hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/TIncrement.java ## @@ -11,7 +11,7 @@ * For increments that are not incrementColumnValue * equivalents. */ -@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2020-04-16") +@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2021-01-12") Review comment: I detect that this code is problematic. According to the [Bad practice (BAD_PRACTICE)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#bad-practice-bad-practice), [Nm: Method names should start with a lower case letter (NM_METHOD_NAMING_CONVENTION)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#nm-method-names-should-start-with-a-lower-case-letter-nm-method-naming-convention). Methods should be verbs, in mixed case with the first letter lowercase, with the first letter of each internal word capitalized. ## File path: hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/TCell.java ## @@ -13,7 +13,7 @@ * the timestamp of a cell to a first-class value, making it easy to take * note of temporal data. Cell is used all the way from HStore up to HTable. */ -@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2020-04-16") +@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2021-01-12") Review comment: I detect that this code is problematic. According to the [Bad practice (BAD_PRACTICE)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#bad-practice-bad-practice), [Nm: Method names should start with a lower case letter (NM_METHOD_NAMING_CONVENTION)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#nm-method-names-should-start-with-a-lower-case-letter-nm-method-naming-convention). Methods should be verbs, in mixed case with the first letter lowercase, with the first letter of each internal word capitalized. ## File path: hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/TAppend.java ## @@ -10,7 +10,7 @@ /** * An Append object is used to specify the parameters for performing the append operation. */ -@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2020-04-16") +@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2021-01-12") Review comment: I detect that this code is problematic. According to the [Bad practice (BAD_PRACTICE)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#bad-practice-bad-practice), [Nm: Method names should start with a lower case letter (NM_METHOD_NAMING_CONVENTION)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#nm-method-names-should-start-with-a-lower-case-letter-nm-method-naming-convention). Methods should be verbs, in mixed case with the first letter lowercase, with the first letter of each internal word capitalized. ## File path: hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/IllegalArgument.java ## @@ -11,7 +11,7 @@ * An IllegalArgument exception indicates an illegal or invalid * argument was passed into a procedure. */ -@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2020-04-16") +@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2021-01-12") Review comment: I detect that this code is problematic. According to the [Bad practice (BAD_PRACTICE)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#bad-practice-bad-practice), [Nm: Method names should start with a lower case letter (NM_METHOD_NAMING_CONVENTION)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#nm-method-names-should-start-with-a-lower-case-letter-nm-method-naming-convention). Methods should be verbs, in mixed case with the first letter lowercase, with the first letter of each internal word capitalized. ## File path: hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/generated/IOError.java ## @@ -12,7 +12,7 @@ * to the Hbase master or an Hbase region server. Also used to return * more general Hbase error conditions. */ -@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2020-04-16") +@javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2021-01-12") Review comment: I detect that this code is problematic. According to the [Bad practice (BAD_PRACTICE)](https://spotbugs.readthedocs.io/en/stable/bugDescriptions.html#bad-practice-bad-practice), [Nm: Method names should start with a lower case letter (NM_METHOD_NAMING_CON
[jira] [Resolved] (HBASE-25559) Terminate threads of oldsources while RS is closing
[ https://issues.apache.org/jira/browse/HBASE-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sun Xin resolved HBASE-25559. - Fix Version/s: 2.4.2 2.3.5 2.2.7 3.0.0-alpha-1 Resolution: Fixed Merged to master and all active branch-2.x. > Terminate threads of oldsources while RS is closing > --- > > Key: HBASE-25559 > URL: https://issues.apache.org/jira/browse/HBASE-25559 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1, 2.2.6, 2.3.4, 2.4.1 >Reporter: Sun Xin >Assignee: Sun Xin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.5, 2.4.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying'
Apache-HBase commented on pull request #2943: URL: https://github.com/apache/hbase/pull/2943#issuecomment-775788321 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 47s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 3m 20s | master passed | | +1 :green_heart: | checkstyle | 1m 4s | master passed | | +1 :green_heart: | spotbugs | 2m 4s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 3m 14s | the patch passed | | +1 :green_heart: | javac | 3m 14s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 19m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 2m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 17s | The patch does not generate ASF License warnings. | | | | 50m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2943 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 2854a1369347 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 0353909bc2 | | Default Java | AdoptOpenJDK-1.8.0_232-b09 | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-server U: hbase-server | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2943/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=3.1.12 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-21674) Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-21674: -- Fix Version/s: 1.7.0 > Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from > thrift1 server) to branch-1 > > > Key: HBASE-21674 > URL: https://issues.apache.org/jira/browse/HBASE-21674 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Yutong Xiao >Priority: Major > Fix For: 1.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Reidddddd commented on pull request #2941: HBASE-21674:Port HBASE-21652 (Refactor ThriftServer making thrift2 server inherited from thrift1 server) to branch-1
Reidd commented on pull request #2941: URL: https://github.com/apache/hbase/pull/2941#issuecomment-775765450 ping @apurtell @busbey > major version 52 is newer than 51, the highest major version supported by this compiler. It is recommended that the compiler be upgraded. is it expected since we are using jdk 7? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg merged pull request #2938: HBASE-25559 Terminate threads of oldsources while RS is closing
ddupg merged pull request #2938: URL: https://github.com/apache/hbase/pull/2938 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ddupg opened a new pull request #2943: HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying'
ddupg opened a new pull request #2943: URL: https://github.com/apache/hbase/pull/2943 …iately without retrying This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
