[GitHub] [hbase] anoopsjohn commented on a change in pull request #1801: [HBASE-24441]CacheConfig details logged at Store open is not really u…
anoopsjohn commented on a change in pull request #1801:
URL: https://github.com/apache/hbase/pull/1801#discussion_r432812650
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -377,7 +377,7 @@ private MemStore getMemstore() {
* @param family The current column family.
*/
protected void createCacheConf(final ColumnFamilyDescriptor family) {
-this.cacheConf = new CacheConfig(conf, family, region.getBlockCache(),
+this.cacheConf = new CacheConfig(conf, family, region.getRegionInfo(),
region.getBlockCache(),
Review comment:
Ya this is some thing we can avoid
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
##
@@ -205,6 +206,7 @@ public CacheConfig(Configuration conf,
ColumnFamilyDescriptor family, BlockCache
this.blockCache = blockCache;
this.byteBuffAllocator = byteBuffAllocator;
LOG.info("Created cacheConfig: " + this + (family == null ? "" : " for
family " + family) +
Review comment:
IMO better we should move this LOG from here to HStore.
protected void createCacheConf(final ColumnFamilyDescriptor family) {
this.cacheConf = new CacheConfig(conf, family, region.getBlockCache(),
region.getRegionServicesForStores().getByteBuffAllocator());
}
After the cacheConf is created. CacheConfig is having toString() with all
details. This way we can avoid passing the RegionInfo/regionName unnecessarily
to CacheConfig just for log.
In Master branch, we are not seeing the CacheConfig log with every
compaction because of a different constructor being used.
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java
##
@@ -245,7 +245,8 @@ public void testDisableCacheDataBlock() throws IOException {
.setBlockCacheEnabled(false)
.build();
-cacheConfig = new CacheConfig(conf, columnFamilyDescriptor, null,
ByteBuffAllocator.HEAP);
Review comment:
All these test changes will be not needed
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
##
@@ -205,6 +206,7 @@ public CacheConfig(Configuration conf,
ColumnFamilyDescriptor family, BlockCache
this.blockCache = blockCache;
this.byteBuffAllocator = byteBuffAllocator;
LOG.info("Created cacheConfig: " + this + (family == null ? "" : " for
family " + family) +
+(hri == null ? "" : " in region " + hri.getRegionNameAsString()) +
" with blockCache=" + blockCache);
Review comment:
Here this blockCache will just give object identifier as no toString()
there. BlockCache will be single object within RS. So IMO there is no need to
log that with every Store create. Anyway if you do other comment, we will
avoid it as CacheConfig#toString() wont be having BlockCache object
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
anoopsjohn commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432812265
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -331,11 +332,11 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
confPrintThreshold = 10;
}
this.parallelPutCountPrintThreshold = confPrintThreshold;
-LOG.info("Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
-+ "parallelPutCountPrintThreshold={}, encoding={}, compression={}",
+LOG.info("Created HStore: Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
++ "parallelPutCountPrintThreshold={}, encoding={}, compression={}
in region {}",
getColumnFamilyName(), memstore.getClass().getSimpleName(),
policyName, verifyBulkLoads,
parallelPutCountPrintThreshold, family.getDataBlockEncoding(),
-family.getCompressionType());
+family.getCompressionType(), getRegionInfo().getRegionNameAsString());
Review comment:
Oh ya encoded name.. In fact I wanted to say , better we can use
RegionInfo#prettyPrint(encodedRegionName).. Sorry missed adding that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24451) Remove the HasThread because the related bug had been fixed since jdk7
[ https://issues.apache.org/jira/browse/HBASE-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120115#comment-17120115 ] Zheng Wang commented on HBASE-24451: Thanks a lot. > Remove the HasThread because the related bug had been fixed since jdk7 > -- > > Key: HBASE-24451 > URL: https://issues.apache.org/jira/browse/HBASE-24451 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > The HasThread was introduced as a wrapper of Thread to workaround a deadlock > bug, and this bug had fixed since jdk7, so I think it is the time to remove > it as we are using jdk8+. > > Related links: > > [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367] > > [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
Apache-HBase commented on pull request #1648: URL: https://github.com/apache/hbase/pull/1648#issuecomment-636275611 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 4s | master passed | | +1 :green_heart: | compile | 3m 12s | master passed | | +1 :green_heart: | shadedjars | 6m 18s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 10s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | +1 :green_heart: | shadedjars | 6m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 4s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 13s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 193m 8s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 5m 59s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 4m 23s | hbase-rest in the patch passed. | | | | 239m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1648 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 52a821c732b2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/testReport/ | | Max. process+thread count | 3232 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
Apache-HBase commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-636274875 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 8m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 10m 38s | branch-1 passed | | +1 :green_heart: | compile | 0m 55s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 1m 4s | branch-1 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 2m 25s | branch-1 passed | | +1 :green_heart: | shadedjars | 3m 50s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 52s | branch-1 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 3m 42s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 39s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 29s | the patch passed | | +1 :green_heart: | compile | 0m 54s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javac | 0m 54s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | javac | 1m 5s | the patch passed | | -1 :x: | checkstyle | 2m 20s | hbase-server: The patch generated 18 new + 245 unchanged - 36 fixed = 263 total (was 281) | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedjars | 4m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 58s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 39s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 53s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 3m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 131m 47s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. | | | | 192m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1755 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 256d68c4cf8b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1755/out/precommit/personality/provided.sh | | git revision | branch-1 / 61fa21e | | Default Java | 1.7.0_262 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_252 /usr/lib/jvm/zulu-7-amd64:1.7.0_262 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/3/artifact/out/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/3/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/3/testReport/ | | Max. process+thread count | 4653 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/3/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1805: HBASE-24467 Backport HBASE-23963: Split TestFromClientSide; it takes …
Apache-HBase commented on pull request #1805: URL: https://github.com/apache/hbase/pull/1805#issuecomment-636270965 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 49s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 9 new or modified test files. | ||| _ branch-2.2 Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 7m 1s | branch-2.2 passed | | +1 :green_heart: | compile | 1m 41s | branch-2.2 passed | | +1 :green_heart: | checkstyle | 2m 33s | branch-2.2 passed | | +1 :green_heart: | shadedjars | 5m 47s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | branch-2.2 passed | | +0 :ok: | spotbugs | 4m 36s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 55s | branch-2.2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 6m 19s | the patch passed | | +1 :green_heart: | compile | 1m 24s | the patch passed | | +1 :green_heart: | javac | 1m 24s | the patch passed | | -1 :x: | checkstyle | 1m 17s | hbase-server: The patch generated 12 new + 0 unchanged - 48 fixed = 12 total (was 48) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 3m 59s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 25m 12s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 2.10.0 or 3.1.2 3.2.1. | | +1 :green_heart: | javadoc | 0m 55s | the patch passed | | +1 :green_heart: | findbugs | 4m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 3m 6s | hbase-common in the patch passed. | | -1 :x: | unit | 166m 19s | hbase-server in the patch failed. | | +1 :green_heart: | asflicense | 1m 8s | The patch does not generate ASF License warnings. | | | | 246m 29s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestFromClientSideWithCoprocessor5 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1805/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1805 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 4cc953455f44 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1805/out/precommit/personality/provided.sh | | git revision | branch-2.2 / 604d7bdfb8 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1805/2/artifact/out/diff-checkstyle-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1805/2/artifact/out/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1805/2/testReport/ | | Max. process+thread count | 5071 (vs. ulimit of 1) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1805/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
Apache-HBase commented on pull request #1648: URL: https://github.com/apache/hbase/pull/1648#issuecomment-636270089 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 31s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 6s | master passed | | +1 :green_heart: | compile | 3m 29s | master passed | | +1 :green_heart: | shadedjars | 5m 40s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 20s | hbase-rest in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 48s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 3m 33s | the patch passed | | +1 :green_heart: | javac | 3m 33s | the patch passed | | +1 :green_heart: | shadedjars | 5m 40s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 50s | hbase-thrift in the patch failed. | | -0 :warning: | javadoc | 0m 21s | hbase-rest in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 59s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 11s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 130m 50s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 4m 37s | hbase-thrift in the patch passed. | | +1 :green_heart: | unit | 3m 21s | hbase-rest in the patch passed. | | | | 176m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1648 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 27f56933c39a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-rest.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-rest.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/testReport/ | | Max. process+thread count | 4474 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. --
[GitHub] [hbase] Apache-HBase commented on pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache-HBase commented on pull request #1806: URL: https://github.com/apache/hbase/pull/1806#issuecomment-636269998 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 43s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 1m 49s | master passed | | +1 :green_heart: | shadedjars | 6m 37s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 1m 53s | the patch passed | | +1 :green_heart: | javac | 1m 53s | the patch passed | | +1 :green_heart: | shadedjars | 6m 45s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 144m 15s | hbase-server in the patch passed. | | | | 175m 43s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1806 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 82338eabb7f2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/testReport/ | | Max. process+thread count | 4533 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24470) Add store file info when log meessages in HStore.
[
https://issues.apache.org/jira/browse/HBASE-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120092#comment-17120092
]
Hudson commented on HBASE-24470:
Results for branch branch-2.3
[build #112 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add store file info when log meessages in HStore.
> -
>
> Key: HBASE-24470
> URL: https://issues.apache.org/jira/browse/HBASE-24470
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: song XinCun
>Assignee: song XinCun
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> Some log message do not have store file info when log, need to add it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24428) Priority compaction for recently split daughter regions
[
https://issues.apache.org/jira/browse/HBASE-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120091#comment-17120091
]
Hudson commented on HBASE-24428:
Results for branch branch-2.3
[build #112 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Priority compaction for recently split daughter regions
> ---
>
> Key: HBASE-24428
> URL: https://issues.apache.org/jira/browse/HBASE-24428
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> We observe that under hotspotting conditions that splitting will proceed very
> slowly and the "_Cannot split region due to reference files being there_" log
> line will be logged excessively. (branch-1 based production.) This is because
> after a region is split it must be compacted before it can be split again.
> Reference files must be replaced by real HFiles, normal housekeeping
> performed during compaction. However if the regionserver is under excessive
> load, its compaction queues may become deep. The daughters of a recently
> split hotspotting region may themselves continue to hotspot and will rapidly
> need to split again. If the scheduled compaction work to remove/replace
> reference files is queued hundreds or thousands of compaction queue elements
> behind current, the recently split daughter regions will not be able to split
> again for a long time and may grow very large, producing additional
> complications (very large regions, very deep replication queues).
> To help avoid this condition we should prioritize the compaction of recently
> split daughter regions. Compaction requests include a {{priority}} field and
> CompactionRequest implements a comparator that sorts by this field. We
> already detect when a compaction request involves a region that has reference
> files, to ensure that it gets selected to be eligible for compaction, but we
> do not seem to prioritize the requests for post-split housekeeping. Split
> work should be placed at the top of the queue. Ensure that this is happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24451) Remove the HasThread because the related bug had been fixed since jdk7
[
https://issues.apache.org/jira/browse/HBASE-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120089#comment-17120089
]
Hudson commented on HBASE-24451:
Results for branch branch-2.3
[build #112 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the HasThread because the related bug had been fixed since jdk7
> --
>
> Key: HBASE-24451
> URL: https://issues.apache.org/jira/browse/HBASE-24451
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> The HasThread was introduced as a wrapper of Thread to workaround a deadlock
> bug, and this bug had fixed since jdk7, so I think it is the time to remove
> it as we are using jdk8+.
>
> Related links:
>
> [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367]
>
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24280) Hadoop2 and Hadoop3 profiles being activated simultaneously causing test failures
[
https://issues.apache.org/jira/browse/HBASE-24280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120093#comment-17120093
]
Hudson commented on HBASE-24280:
Results for branch branch-2.3
[build #112 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Hadoop2 and Hadoop3 profiles being activated simultaneously causing test
> failures
> -
>
> Key: HBASE-24280
> URL: https://issues.apache.org/jira/browse/HBASE-24280
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: HBASE-24280.master.001.patch,
> TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml
>
>
> [~ndimiduk] pointed out that, after this change went in, TestSecureRESTServer
> started failing with Hadoop3 on branch-2.3
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/56/
> Of course, I ran this with 1.8.0_241 and Maven 3.6.33 and it passed :) {{mvn
> clean package -Dtest=TestSecureRESTServer -Dhadoop.profile=3.0
> -DfailIfNoTests=false}}
> FYI [~stoty] in case you can repro a failure and want to dig in. Feel free to
> re-assign.
> It looks like we didn't have a nightly run of branch-2.2 due to docker
> container build issues. Will be interesting to see if it fails there. It did
> not fail the master nightly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22287) inifinite retries on failed server in RSProcedureDispatcher
[
https://issues.apache.org/jira/browse/HBASE-22287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120094#comment-17120094
]
Hudson commented on HBASE-22287:
Results for branch branch-2.3
[build #112 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/112/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> inifinite retries on failed server in RSProcedureDispatcher
> ---
>
> Key: HBASE-22287
> URL: https://issues.apache.org/jira/browse/HBASE-22287
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> We observed this recently on some cluster, I'm still investigating the root
> cause however seems like the retries should have special handling for this
> exception; and separately probably a cap on number of retries
> {noformat}
> 2019-04-20 04:24:27,093 WARN [RSProcedureDispatcher-pool4-t1285]
> procedure.RSProcedureDispatcher: request to server ,17020,1555742560432
> failed due to java.io.IOException: Call to :17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: :17020, try=26603, retrying...
> {noformat}
> The corresponding worker is stuck
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache-HBase commented on pull request #1806: URL: https://github.com/apache/hbase/pull/1806#issuecomment-636268338 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 15s | master passed | | +1 :green_heart: | compile | 2m 1s | master passed | | +1 :green_heart: | shadedjars | 5m 47s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 2m 6s | the patch passed | | +1 :green_heart: | javac | 2m 6s | the patch passed | | +1 :green_heart: | shadedjars | 5m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 126m 17s | hbase-server in the patch passed. | | | | 157m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1806 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 93fef3dd42e5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/testReport/ | | Max. process+thread count | 4565 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24436) The store file open and close thread pool should be shared at the region level
[ https://issues.apache.org/jira/browse/HBASE-24436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120083#comment-17120083 ] Zheng Wang commented on HBASE-24436: bq. IMO, the goal should be load all stores as fast as possible. To be clear, the goal I said above not meaning use more threads, so if we have 10 fimalies, we will start 10 threads totally no matter in region level or store level. > The store file open and close thread pool should be shared at the region level > -- > > Key: HBASE-24436 > URL: https://issues.apache.org/jira/browse/HBASE-24436 > Project: HBase > Issue Type: Improvement >Reporter: Junhong Xu >Assignee: Junhong Xu >Priority: Minor > > For now, we provide threads per column family evenly in general, but there > are some cases that some column families have much more store files than > others( maybe that's the life, right? ). So in that case, some Stores have > beed done quickly while others are struggling.We should share the thread pool > at the region level in case of data skew. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24428) Priority compaction for recently split daughter regions
[
https://issues.apache.org/jira/browse/HBASE-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120072#comment-17120072
]
Hudson commented on HBASE-24428:
Results for branch branch-2
[build #2683 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Priority compaction for recently split daughter regions
> ---
>
> Key: HBASE-24428
> URL: https://issues.apache.org/jira/browse/HBASE-24428
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> We observe that under hotspotting conditions that splitting will proceed very
> slowly and the "_Cannot split region due to reference files being there_" log
> line will be logged excessively. (branch-1 based production.) This is because
> after a region is split it must be compacted before it can be split again.
> Reference files must be replaced by real HFiles, normal housekeeping
> performed during compaction. However if the regionserver is under excessive
> load, its compaction queues may become deep. The daughters of a recently
> split hotspotting region may themselves continue to hotspot and will rapidly
> need to split again. If the scheduled compaction work to remove/replace
> reference files is queued hundreds or thousands of compaction queue elements
> behind current, the recently split daughter regions will not be able to split
> again for a long time and may grow very large, producing additional
> complications (very large regions, very deep replication queues).
> To help avoid this condition we should prioritize the compaction of recently
> split daughter regions. Compaction requests include a {{priority}} field and
> CompactionRequest implements a comparator that sorts by this field. We
> already detect when a compaction request involves a region that has reference
> files, to ensure that it gets selected to be eligible for compaction, but we
> do not seem to prioritize the requests for post-split housekeeping. Split
> work should be placed at the top of the queue. Ensure that this is happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24280) Hadoop2 and Hadoop3 profiles being activated simultaneously causing test failures
[
https://issues.apache.org/jira/browse/HBASE-24280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120074#comment-17120074
]
Hudson commented on HBASE-24280:
Results for branch branch-2
[build #2683 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Hadoop2 and Hadoop3 profiles being activated simultaneously causing test
> failures
> -
>
> Key: HBASE-24280
> URL: https://issues.apache.org/jira/browse/HBASE-24280
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: HBASE-24280.master.001.patch,
> TEST-org.apache.hadoop.hbase.rest.TestSecureRESTServer.xml
>
>
> [~ndimiduk] pointed out that, after this change went in, TestSecureRESTServer
> started failing with Hadoop3 on branch-2.3
> https://builds.apache.org/job/HBase%20Nightly/job/branch-2.3/56/
> Of course, I ran this with 1.8.0_241 and Maven 3.6.33 and it passed :) {{mvn
> clean package -Dtest=TestSecureRESTServer -Dhadoop.profile=3.0
> -DfailIfNoTests=false}}
> FYI [~stoty] in case you can repro a failure and want to dig in. Feel free to
> re-assign.
> It looks like we didn't have a nightly run of branch-2.2 due to docker
> container build issues. Will be interesting to see if it fails there. It did
> not fail the master nightly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22287) inifinite retries on failed server in RSProcedureDispatcher
[
https://issues.apache.org/jira/browse/HBASE-22287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120075#comment-17120075
]
Hudson commented on HBASE-22287:
Results for branch branch-2
[build #2683 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> inifinite retries on failed server in RSProcedureDispatcher
> ---
>
> Key: HBASE-22287
> URL: https://issues.apache.org/jira/browse/HBASE-22287
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> We observed this recently on some cluster, I'm still investigating the root
> cause however seems like the retries should have special handling for this
> exception; and separately probably a cap on number of retries
> {noformat}
> 2019-04-20 04:24:27,093 WARN [RSProcedureDispatcher-pool4-t1285]
> procedure.RSProcedureDispatcher: request to server ,17020,1555742560432
> failed due to java.io.IOException: Call to :17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: :17020, try=26603, retrying...
> {noformat}
> The corresponding worker is stuck
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24470) Add store file info when log meessages in HStore.
[
https://issues.apache.org/jira/browse/HBASE-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120073#comment-17120073
]
Hudson commented on HBASE-24470:
Results for branch branch-2
[build #2683 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add store file info when log meessages in HStore.
> -
>
> Key: HBASE-24470
> URL: https://issues.apache.org/jira/browse/HBASE-24470
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: song XinCun
>Assignee: song XinCun
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> Some log message do not have store file info when log, need to add it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24451) Remove the HasThread because the related bug had been fixed since jdk7
[
https://issues.apache.org/jira/browse/HBASE-24451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120071#comment-17120071
]
Hudson commented on HBASE-24451:
Results for branch branch-2
[build #2683 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2683/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the HasThread because the related bug had been fixed since jdk7
> --
>
> Key: HBASE-24451
> URL: https://issues.apache.org/jira/browse/HBASE-24451
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> The HasThread was introduced as a wrapper of Thread to workaround a deadlock
> bug, and this bug had fixed since jdk7, so I think it is the time to remove
> it as we are using jdk8+.
>
> Related links:
>
> [https://issues.apache.org/jira/browse/HBASE-4367|https://slack-redir.net/link?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHBASE-4367]
>
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6915621|https://slack-redir.net/link?url=https%3A%2F%2Fbugs.java.com%2Fbugdatabase%2Fview_bug.do%3Fbug_id%3D6915621]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1648: HBASE-8458 Support for batch version of checkAndMutate()
Apache-HBase commented on pull request #1648: URL: https://github.com/apache/hbase/pull/1648#issuecomment-636257674 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | checkstyle | 3m 0s | master passed | | +1 :green_heart: | spotbugs | 9m 15s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 49s | the patch passed | | +1 :green_heart: | checkstyle | 0m 8s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 0m 34s | hbase-client: The patch generated 0 new + 130 unchanged - 5 fixed = 130 total (was 135) | | -0 :warning: | checkstyle | 1m 15s | hbase-server: The patch generated 1 new + 67 unchanged - 0 fixed = 68 total (was 67) | | +1 :green_heart: | checkstyle | 0m 49s | The patch passed checkstyle in hbase-thrift | | +1 :green_heart: | checkstyle | 0m 15s | The patch passed checkstyle in hbase-rest | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 43s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 3m 25s | the patch passed | | +1 :green_heart: | spotbugs | 11m 34s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 58s | The patch does not generate ASF License warnings. | | | | 62m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1648 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 9a53f50acb96 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-rest U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1648/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
Apache-HBase commented on pull request #1786: URL: https://github.com/apache/hbase/pull/1786#issuecomment-636256219 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 42s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 16s | master passed | | +1 :green_heart: | compile | 3m 21s | master passed | | +1 :green_heart: | shadedjars | 7m 3s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | root in master failed. | | -0 :warning: | javadoc | 0m 18s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 50s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 3s | the patch passed | | +1 :green_heart: | compile | 3m 36s | the patch passed | | +1 :green_heart: | javac | 3m 36s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 43s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 15s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 273m 8s | root in the patch passed. | | | | 312m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1786 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 89047dded43f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06949ff6a6 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/testReport/ | | Max. process+thread count | 4855 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache-HBase commented on pull request #1806: URL: https://github.com/apache/hbase/pull/1806#issuecomment-636254732 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | checkstyle | 1m 23s | master passed | | +1 :green_heart: | spotbugs | 6m 2s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 11s | The patch passed checkstyle in hbase-protocol-shaded | | +1 :green_heart: | checkstyle | 1m 3s | hbase-server: The patch generated 0 new + 66 unchanged - 4 fixed = 66 total (was 70) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 10s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 1m 33s | the patch passed | | +1 :green_heart: | spotbugs | 5m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 43m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1806 | | Optional Tests | dupname asflicense cc hbaseprotoc prototool spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 2aa027b4c944 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 78fce0f333 | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] sguggilam commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-636253821 @bharathv pushed the latest changes incorporating review comments, please check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-native-client] phrocker commented on a change in pull request #2: HBASE-24400: Fixup cmake infrastructure to allow dependencies to be built locally
phrocker commented on a change in pull request #2:
URL: https://github.com/apache/hbase-native-client/pull/2#discussion_r432795101
##
File path: cmake/DownloadFolly.cmake
##
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+## Download facebook's folly library.
+## SOURCE_DIR is typically the cmake source directory
+## BINARY_DIR is the build directory, typically 'build'
+
+function(download_folly SOURCE_DIR BINARY_DIR)
+
+
+ ExternalProject_Add(
+ facebook-folly-proj
+ GIT_REPOSITORY "https://github.com/facebook/folly.git";
+ GIT_TAG "v2020.05.18.00"
+ SOURCE_DIR "${BINARY_DIR}/dependencies/facebook-folly-proj-src"
+ CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
Review comment:
Totally agree. I'm going to submit a separate PR where it should have
just the cmake changes without changing dependencies. Since I already have
these changes that should prep a separate PR for that change set. For now I'm
working through compilation issues on the older folly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] songxincun commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
songxincun commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432792487
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1192,8 +1198,10 @@ public StoreFileWriter createWriterInTmp(long
maxKeyCount, Compression.Algorithm
if (shouldCacheDataOnWrite) {
writerCacheConf.enableCacheOnWrite();
if (!cacheOnWriteLogged) {
- LOG.info("For Store {} , cacheDataOnWrite is true, hence enabled
cacheOnWrite for " +
-"Index blocks and Bloom filter blocks", getColumnFamilyName());
+ LOG.info("For Store {} in region {}, " +
Review comment:
> No need of 'Store' or 'region'... its obvious that we're talking store
and region. Having these works in there bulk up logging and make it harder to
read/parse.
Got it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] songxincun commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
songxincun commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432792434
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -331,11 +332,11 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
confPrintThreshold = 10;
}
this.parallelPutCountPrintThreshold = confPrintThreshold;
-LOG.info("Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
-+ "parallelPutCountPrintThreshold={}, encoding={}, compression={}",
+LOG.info("Created HStore: Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
Review comment:
> Why have 'HStore:' in here? Instead of 'Created HStore: Store=', just
have 'Created '.
Got it
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -331,11 +332,11 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
confPrintThreshold = 10;
}
this.parallelPutCountPrintThreshold = confPrintThreshold;
-LOG.info("Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
-+ "parallelPutCountPrintThreshold={}, encoding={}, compression={}",
+LOG.info("Created HStore: Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
++ "parallelPutCountPrintThreshold={}, encoding={}, compression={}
in region {}",
getColumnFamilyName(), memstore.getClass().getSimpleName(),
policyName, verifyBulkLoads,
parallelPutCountPrintThreshold, family.getDataBlockEncoding(),
-family.getCompressionType());
+family.getCompressionType(), getRegionInfo().getRegionNameAsString());
Review comment:
> Encoded name, not full name.
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] songxincun commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
songxincun commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432792040
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -276,7 +276,8 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
// used by ScanQueryMatcher
long timeToPurgeDeletes =
Math.max(conf.getLong("hbase.hstore.time.to.purge.deletes", 0), 0);
-LOG.trace("Time to purge deletes set to {}ms in store {}",
timeToPurgeDeletes, this);
+LOG.trace("Time to purge deletes set to {}ms in store {} in region {}",
timeToPurgeDeletes,
Review comment:
Good suggestions, I will do more work on this, thank you for all your
reviews
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24474) Rename LocalRegion to MasterRegion
[ https://issues.apache.org/jira/browse/HBASE-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120034#comment-17120034 ] Michael Stack commented on HBASE-24474: --- I like this idea. > Rename LocalRegion to MasterRegion > -- > > Key: HBASE-24474 > URL: https://issues.apache.org/jira/browse/HBASE-24474 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0 > > > This is a suggestion by [~ndimiduk] when reviewing the PR for HBASE-24408. > https://github.com/apache/hbase/pull/1753#discussion_r432783115 > I think this can make the code less confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24343) Document how to configure the http request log
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24343: - Fix Version/s: 2.3.0 3.0.0-alpha-1 > Document how to configure the http request log > -- > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.3.0 >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24343) Document how to configure the http request log
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24343. -- Resolution: Fixed > Document how to configure the http request log > -- > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.3.0 >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #1807: HBASE-24343 Document how to configure the http request log
ndimiduk commented on pull request #1807: URL: https://github.com/apache/hbase/pull/1807#issuecomment-636243383 Thanks @saintstack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1809: Backport "HBASE-24343 Document how to configure the http request log" to branch-2.3
ndimiduk merged pull request #1809: URL: https://github.com/apache/hbase/pull/1809 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #1809: Backport "HBASE-24343 Document how to configure the http request log" to branch-2.3
ndimiduk opened a new pull request #1809: URL: https://github.com/apache/hbase/pull/1809 Signed-off-by: Michael Stack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1808: Backport "HBASE-24343 Document how to configure the http request log" to branch-2
ndimiduk merged pull request #1808: URL: https://github.com/apache/hbase/pull/1808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk opened a new pull request #1808: Backport "HBASE-24343 Document how to configure the http request log" to branch-2
ndimiduk opened a new pull request #1808: URL: https://github.com/apache/hbase/pull/1808 Signed-off-by: Michael Stack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24471) The way we bootstrap meta table is confusing
[ https://issues.apache.org/jira/browse/HBASE-24471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-24471: -- Fix Version/s: (was: 2.3.0) 2.4.0 > The way we bootstrap meta table is confusing > > > Key: HBASE-24471 > URL: https://issues.apache.org/jira/browse/HBASE-24471 > Project: HBase > Issue Type: Improvement > Components: master, meta, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > I used to thought the work is only done in the InitMetaProcedure but in fact > we have several pieces of code related to this which are placed in different > classes. > In HRegionServer.initializeFileSystem, we will test whether we are HMaster. > If so, we will call FSTableDescriptors.tryUpdateMetaTableDescriptor. And in > MasterFileSystem.checkRootDir, we will check if meta region is available. If > not, we will bootstrap it, and it will also call > FSTableDescriptors.tryUpdateMetaTableDescriptor. > The problem here is that, HRegionServer.initializeFileSystem is called in the > constructor of HRegionServer, which means it will be executed before > MasterFileSystem.checkRootDir, as we will only create the MasterFileSystem > for active master... > Ideally I prefer we move all the bootstrap code into InitMetaProcedure, which > will be much cleaner. If this can not be done, at least we could move the > code into MasterFileSystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24474) Rename LocalRegion to MasterRegion
Duo Zhang created HBASE-24474: - Summary: Rename LocalRegion to MasterRegion Key: HBASE-24474 URL: https://issues.apache.org/jira/browse/HBASE-24474 Project: HBase Issue Type: Improvement Components: master Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0-alpha-1, 2.3.0 This is a suggestion by [~ndimiduk] when reviewing the PR for HBASE-24408. https://github.com/apache/hbase/pull/1753#discussion_r432783115 I think this can make the code less confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-24474) Rename LocalRegion to MasterRegion
[ https://issues.apache.org/jira/browse/HBASE-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24474 started by Duo Zhang. - > Rename LocalRegion to MasterRegion > -- > > Key: HBASE-24474 > URL: https://issues.apache.org/jira/browse/HBASE-24474 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.3.0 > > > This is a suggestion by [~ndimiduk] when reviewing the PR for HBASE-24408. > https://github.com/apache/hbase/pull/1753#discussion_r432783115 > I think this can make the code less confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #1753: HBASE-24408 Introduce a general 'local region' to store data on master
Apache9 commented on a change in pull request #1753:
URL: https://github.com/apache/hbase/pull/1753#discussion_r432786558
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/store/LocalStore.java
##
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.store;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.regionserver.HRegion.FlushResult;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Used for storing data at master side. The data will be stored in a {@link
LocalRegion}.
+ */
+@InterfaceAudience.Private
+public final class LocalStore {
Review comment:
https://issues.apache.org/jira/browse/HBASE-24474
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1753: HBASE-24408 Introduce a general 'local region' to store data on master
Apache9 commented on a change in pull request #1753:
URL: https://github.com/apache/hbase/pull/1753#discussion_r432786145
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/store/LocalStore.java
##
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.store;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.regionserver.HRegion.FlushResult;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Used for storing data at master side. The data will be stored in a {@link
LocalRegion}.
+ */
+@InterfaceAudience.Private
+public final class LocalStore {
Review comment:
Sounds fine.
Let me open a new issue for this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] saintstack commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
saintstack commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432785781
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -331,11 +332,11 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
confPrintThreshold = 10;
}
this.parallelPutCountPrintThreshold = confPrintThreshold;
-LOG.info("Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
-+ "parallelPutCountPrintThreshold={}, encoding={}, compression={}",
+LOG.info("Created HStore: Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
Review comment:
Why have 'HStore:' in here? Instead of 'Created HStore: Store=', just
have 'Created '.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -276,7 +276,8 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
// used by ScanQueryMatcher
long timeToPurgeDeletes =
Math.max(conf.getLong("hbase.hstore.time.to.purge.deletes", 0), 0);
-LOG.trace("Time to purge deletes set to {}ms in store {}",
timeToPurgeDeletes, this);
+LOG.trace("Time to purge deletes set to {}ms in store {} in region {}",
timeToPurgeDeletes,
Review comment:
I like what @anoopsjohn says. toString could be
encodedRegionName/store-name (region name is too much -- could be anything
given it includes start key... could be a spew of binary characters)
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -331,11 +332,11 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
confPrintThreshold = 10;
}
this.parallelPutCountPrintThreshold = confPrintThreshold;
-LOG.info("Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
-+ "parallelPutCountPrintThreshold={}, encoding={}, compression={}",
+LOG.info("Created HStore: Store={}, memstore type={}, storagePolicy={},
verifyBulkLoads={}, "
++ "parallelPutCountPrintThreshold={}, encoding={}, compression={}
in region {}",
getColumnFamilyName(), memstore.getClass().getSimpleName(),
policyName, verifyBulkLoads,
parallelPutCountPrintThreshold, family.getDataBlockEncoding(),
-family.getCompressionType());
+family.getCompressionType(), getRegionInfo().getRegionNameAsString());
Review comment:
Encoded name, not full name.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -276,7 +276,8 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
// used by ScanQueryMatcher
long timeToPurgeDeletes =
Math.max(conf.getLong("hbase.hstore.time.to.purge.deletes", 0), 0);
-LOG.trace("Time to purge deletes set to {}ms in store {}",
timeToPurgeDeletes, this);
+LOG.trace("Time to purge deletes set to {}ms in store {} in region {}",
timeToPurgeDeletes,
Review comment:
Otherwise I'd comment that no need of words 'store' or 'region' in log
message; no need of these qualifiers.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -1192,8 +1198,10 @@ public StoreFileWriter createWriterInTmp(long
maxKeyCount, Compression.Algorithm
if (shouldCacheDataOnWrite) {
writerCacheConf.enableCacheOnWrite();
if (!cacheOnWriteLogged) {
- LOG.info("For Store {} , cacheDataOnWrite is true, hence enabled
cacheOnWrite for " +
-"Index blocks and Bloom filter blocks", getColumnFamilyName());
+ LOG.info("For Store {} in region {}, " +
Review comment:
No need of 'Store' or 'region'... its obvious that we're talking store
and region. Having these works in there bulk up logging and make it harder to
read/parse.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache9 commented on a change in pull request #1806:
URL: https://github.com/apache/hbase/pull/1806#discussion_r432786009
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -64,41 +72,68 @@ public TableOperationType getTableOperationType() {
return TableOperationType.CREATE;
}
+ private static void writeFsLayout(Path rootDir, Configuration conf) throws
IOException {
+LOG.info("BOOTSTRAP: creating hbase:meta region");
+FileSystem fs = rootDir.getFileSystem(conf);
+Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
+if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+ LOG.warn("Can not delete partial created meta table, continue...");
+}
+// Bootstrapping, make sure blockcache is off. Else, one will be
+// created here in bootstrap and it'll need to be cleaned up. Better to
+// not make it in first place. Turn off block caching for bootstrap.
+// Enable after.
+FSTableDescriptors.tryUpdateMetaTableDescriptor(conf, fs, rootDir,
+ builder -> builder.setRegionReplication(
+conf.getInt(HConstants.META_REPLICAS_NUM,
HConstants.DEFAULT_META_REPLICA_NUM)));
+TableDescriptor metaDescriptor = new
FSTableDescriptors(conf).get(TableName.META_TABLE_NAME);
+HRegion
+ .createHRegion(RegionInfoBuilder.FIRST_META_REGIONINFO, rootDir, conf,
metaDescriptor, null)
+ .close();
+ }
+
@Override
protected Flow executeFromState(MasterProcedureEnv env, InitMetaState state)
- throws ProcedureSuspendedException, ProcedureYieldException,
InterruptedException {
+throws ProcedureSuspendedException, ProcedureYieldException,
InterruptedException {
LOG.debug("Execute {}", this);
-switch (state) {
- case INIT_META_ASSIGN_META:
-LOG.info("Going to assign meta");
-addChildProcedure(env.getAssignmentManager()
-
.createAssignProcedures(Arrays.asList(RegionInfoBuilder.FIRST_META_REGIONINFO)));
-setNextState(InitMetaState.INIT_META_CREATE_NAMESPACES);
-return Flow.HAS_MORE_STATE;
- case INIT_META_CREATE_NAMESPACES:
-LOG.info("Going to create {} and {} namespaces", DEFAULT_NAMESPACE,
SYSTEM_NAMESPACE);
-try {
+try {
+ switch (state) {
+case INIT_META_WRITE_FS_LAYOUT:
+ Configuration conf = env.getMasterConfiguration();
+ Path rootDir = CommonFSUtils.getRootDir(conf);
+ writeFsLayout(rootDir, conf);
+ setNextState(InitMetaState.INIT_META_ASSIGN_META);
+ return Flow.HAS_MORE_STATE;
+case INIT_META_ASSIGN_META:
+ LOG.info("Going to assign meta");
+ addChildProcedure(env.getAssignmentManager()
Review comment:
You misunderstood the goal here...
The procedure is 'InitMetaProcedure', not 'InitRootDirProcedure', it is just
for initializaing meta table. The code for initializing the root dir is still
in MasterFileSystem, what you have reviewed above...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache9 commented on a change in pull request #1806:
URL: https://github.com/apache/hbase/pull/1806#discussion_r432785688
##
File path:
hbase-protocol-shaded/src/main/protobuf/server/master/MasterProcedure.proto
##
@@ -489,8 +489,9 @@ message ReopenTableRegionsStateData {
}
enum InitMetaState {
- INIT_META_ASSIGN_META = 1;
- INIT_META_CREATE_NAMESPACES = 2;
+ INIT_META_WRITE_FS_LAYOUT = 1;
Review comment:
I tried to do this in the patch, i.e, adding 'INIT_META_WRITE_FS_LAYOUT
= 3', but finally I found that it is not necessary, as changing the order or
adding a new enum will both break the compatibility, so let's just kep the code
clean, i.e, the order in the enum is the order when executing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk merged pull request #1807: HBASE-24343 Document how to configure the http request log
ndimiduk merged pull request #1807: URL: https://github.com/apache/hbase/pull/1807 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
ndimiduk commented on a change in pull request #1806:
URL: https://github.com/apache/hbase/pull/1806#discussion_r432767573
##
File path:
hbase-protocol-shaded/src/main/protobuf/server/master/MasterProcedure.proto
##
@@ -489,8 +489,9 @@ message ReopenTableRegionsStateData {
}
enum InitMetaState {
- INIT_META_ASSIGN_META = 1;
- INIT_META_CREATE_NAMESPACES = 2;
+ INIT_META_WRITE_FS_LAYOUT = 1;
Review comment:
Why not preserve the existing ordinals? The state machine implemented
over this enum can control the transitions however it sees fit, and you'll can
keep serialized data that much more backwards-compatible.
Unless these existing states have new meanings in the new paradigm...
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
##
@@ -245,61 +238,56 @@ public ClusterId getClusterId() {
* directory with necessary bootup files).
*/
private Path checkRootDir(final Path rd, final Configuration c, final
FileSystem fs)
- throws IOException {
+throws IOException {
// If FS is in safe mode wait till out of it.
FSUtils.waitOnSafeMode(c, c.getInt(HConstants.THREAD_WAKE_FREQUENCY, 10 *
1000));
// Filesystem is good. Go ahead and check for hbase.rootdir.
+FileStatus status;
+try {
+ status = fs.getFileStatus(rd);
+} catch (FileNotFoundException e) {
+ status = null;
+}
try {
- if (!fs.exists(rd)) {
-fs.mkdirs(rd);
+ if (status == null) {
+if (!fs.mkdirs(rd)) {
+ throw new IOException("Can not create root dir: " + rd);
+}
// DFS leaves safe mode with 0 DNs when there are 0 blocks.
// We used to handle this by checking the current DN count and waiting
until
// it is nonzero. With security, the check for datanode count doesn't
work --
// it is a privileged op. So instead we adopt the strategy of the
jobtracker
// and simply retry file creation during bootstrap indefinitely. As
soon as
// there is one datanode it will succeed. Permission problems should
have
// already been caught by mkdirs above.
-FSUtils.setVersion(fs, rd, c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
- 10 * 1000), c.getInt(HConstants.VERSION_FILE_WRITE_ATTEMPTS,
-HConstants.DEFAULT_VERSION_FILE_WRITE_ATTEMPTS));
+FSUtils.setVersion(fs, rd, c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
10 * 1000), c.getInt(
+ HConstants.VERSION_FILE_WRITE_ATTEMPTS,
HConstants.DEFAULT_VERSION_FILE_WRITE_ATTEMPTS));
} else {
-if (!fs.isDirectory(rd)) {
- throw new IllegalArgumentException(rd.toString() + " is not a
directory");
+if (!status.isDirectory()) {
+ throw new IllegalArgumentException(rd + " is not a directory");
Review comment:
How about `"Configured '" + HConstants.HBASE_DIR + "' + rd + " is not a
directory."` ?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -64,41 +72,68 @@ public TableOperationType getTableOperationType() {
return TableOperationType.CREATE;
}
+ private static void writeFsLayout(Path rootDir, Configuration conf) throws
IOException {
+LOG.info("BOOTSTRAP: creating hbase:meta region");
+FileSystem fs = rootDir.getFileSystem(conf);
+Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
+if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+ LOG.warn("Can not delete partial created meta table, continue...");
+}
+// Bootstrapping, make sure blockcache is off. Else, one will be
+// created here in bootstrap and it'll need to be cleaned up. Better to
+// not make it in first place. Turn off block caching for bootstrap.
+// Enable after.
+FSTableDescriptors.tryUpdateMetaTableDescriptor(conf, fs, rootDir,
+ builder -> builder.setRegionReplication(
+conf.getInt(HConstants.META_REPLICAS_NUM,
HConstants.DEFAULT_META_REPLICA_NUM)));
+TableDescriptor metaDescriptor = new
FSTableDescriptors(conf).get(TableName.META_TABLE_NAME);
+HRegion
+ .createHRegion(RegionInfoBuilder.FIRST_META_REGIONINFO, rootDir, conf,
metaDescriptor, null)
+ .close();
+ }
+
@Override
protected Flow executeFromState(MasterProcedureEnv env, InitMetaState state)
- throws ProcedureSuspendedException, ProcedureYieldException,
InterruptedException {
+throws ProcedureSuspendedException, ProcedureYieldException,
InterruptedException {
LOG.debug("Execute {}", this);
-switch (state) {
- case INIT_META_ASSIGN_META:
-LOG.info("Going to assign meta");
-addChildProcedure(env.getAssignmentManager()
-
.createAssignProcedures(Arrays.asList(RegionInfoBuilder.FIRST_META_REGIONINFO)));
-setNextState(InitMetaState.INIT_META_CREATE_NAMESPACES);
-return
[GitHub] [hbase] saintstack commented on pull request #1805: HBASE-24467 Backport HBASE-23963: Split TestFromClientSide; it takes …
saintstack commented on pull request #1805: URL: https://github.com/apache/hbase/pull/1805#issuecomment-636238692 Rerunning while @infraio is sleeping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #1753: HBASE-24408 Introduce a general 'local region' to store data on master
ndimiduk commented on a change in pull request #1753:
URL: https://github.com/apache/hbase/pull/1753#discussion_r432783115
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/store/LocalStore.java
##
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.master.store;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.Server;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.regionserver.HRegion.FlushResult;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.yetus.audience.InterfaceAudience;
+
+import
org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
+
+/**
+ * Used for storing data at master side. The data will be stored in a {@link
LocalRegion}.
+ */
+@InterfaceAudience.Private
+public final class LocalStore {
Review comment:
This class name confuses me. I see now that it uses "store" as a legacy
of what was earlier called the "ProcedureStore". The confusion is this: our
Regions also have an internal structure called a "Store".
Maybe `o.a.h.h.master.store` should now be called `o.a.h.h.master.region`.
This class is apparently just a delegate to the region instance. Maybe we can
rename `LocalRegion` to `MasterRegion`, do away with this `LocalStore`, replace
it with a `MasterRegionFactory`, and let the caller invoke methods directly on
the `MasterRegion`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
bharathv commented on a change in pull request #1755:
URL: https://github.com/apache/hbase/pull/1755#discussion_r432774809
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -1995,97 +1995,100 @@ private void unassign(final HRegionInfo region,
}
return;
}
+ long sleepTime = 0;
try {
// Send CLOSE RPC
-if (serverManager.sendRegionClose(server, region,
- versionOfClosingNode, dest, transitionInZK)) {
- LOG.debug("Sent CLOSE to " + server + " for region " +
-region.getRegionNameAsString());
+if (serverManager.sendRegionClose(server, region,
versionOfClosingNode, dest,
+ transitionInZK)) {
+ LOG.debug("Sent CLOSE to " + server + " for region " +
region.getRegionNameAsString());
if (useZKForAssignment && !transitionInZK && state != null) {
// Retry to make sure the region is
// closed so as to avoid double assignment.
-unassign(region, state, versionOfClosingNode,
- dest, transitionInZK, src);
+unassign(region, state, versionOfClosingNode, dest,
transitionInZK, src);
}
return;
}
// This never happens. Currently regionserver close always return true.
// Todo; this can now happen (0.96) if there is an exception in a
coprocessor
-LOG.warn("Server " + server + " region CLOSE RPC returned false for " +
- region.getRegionNameAsString());
- } catch (Throwable t) {
-long sleepTime = 0;
+LOG.warn("Server " + server + " region CLOSE RPC returned false for "
++ region.getRegionNameAsString());
+ } catch (Throwable t) {
Configuration conf = this.server.getConfiguration();
if (t instanceof RemoteException) {
- t = ((RemoteException)t).unwrapRemoteException();
+ t = ((RemoteException) t).unwrapRemoteException();
}
boolean logRetries = true;
-if (t instanceof RegionServerAbortedException
-|| t instanceof RegionServerStoppedException
+if (t instanceof RegionServerAbortedException || t instanceof
RegionServerStoppedException
Review comment:
nit: (t instanceof RegionServerAbortedException) is redundant (per
static analysis check in the IDE)...
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
##
@@ -598,6 +601,68 @@ public void testCloseFailed() throws Exception {
}
}
+ /**
+ * This tests region close with exponential backoff
+ */
+ @Test(timeout = 6)
+ public void testCloseRegionWithExponentialBackOff() throws Exception {
+String table = "testCloseRegionWithExponentialBackOff";
Review comment:
nit: We use TestTableName.getTableName() that automatically gets the
TableName with the running test name.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
##
@@ -1995,97 +1995,100 @@ private void unassign(final HRegionInfo region,
}
return;
}
+ long sleepTime = 0;
try {
// Send CLOSE RPC
-if (serverManager.sendRegionClose(server, region,
- versionOfClosingNode, dest, transitionInZK)) {
- LOG.debug("Sent CLOSE to " + server + " for region " +
-region.getRegionNameAsString());
+if (serverManager.sendRegionClose(server, region,
versionOfClosingNode, dest,
+ transitionInZK)) {
+ LOG.debug("Sent CLOSE to " + server + " for region " +
region.getRegionNameAsString());
if (useZKForAssignment && !transitionInZK && state != null) {
// Retry to make sure the region is
// closed so as to avoid double assignment.
-unassign(region, state, versionOfClosingNode,
- dest, transitionInZK, src);
+unassign(region, state, versionOfClosingNode, dest,
transitionInZK, src);
}
return;
}
// This never happens. Currently regionserver close always return true.
// Todo; this can now happen (0.96) if there is an exception in a
coprocessor
-LOG.warn("Server " + server + " region CLOSE RPC returned false for " +
- region.getRegionNameAsString());
- } catch (Throwable t) {
-long sleepTime = 0;
+LOG.warn("Server " + server + " region CLOSE RPC returned false for "
++ region.getRegionNameAsString());
+ } catch (Throwable t) {
Configuration conf = this.server.getConfiguration();
if (t instanceof RemoteException) {
- t = ((RemoteException)t).unwrapRemoteException();
+ t = ((RemoteException) t).unwrapRemoteException();
}
boolean logRetries = true;
-if (t instanceof RegionServerAbortedException
-
[GitHub] [hbase] Apache-HBase commented on pull request #1793: HBASE-23941 : FilterBy operator support in get_slowlog_responses API
Apache-HBase commented on pull request #1793: URL: https://github.com/apache/hbase/pull/1793#issuecomment-636219779 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 8s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 45s | master passed | | +1 :green_heart: | compile | 3m 4s | master passed | | +1 :green_heart: | shadedjars | 6m 21s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 13s | root in master failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 27s | the patch passed | | +1 :green_heart: | compile | 3m 4s | the patch passed | | +1 :green_heart: | javac | 3m 4s | the patch passed | | +1 :green_heart: | shadedjars | 6m 49s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 14s | root in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 214m 29s | root in the patch failed. | | | | 250m 49s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1793 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a821839371d8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / bda2094ae5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/testReport/ | | Max. process+thread count | 2854 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24429) Flaky test, TestFromClientSide5.testScanMetrics[2]
[
https://issues.apache.org/jira/browse/HBASE-24429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119990#comment-17119990
]
Huaxiang Sun commented on HBASE-24429:
--
One of the latest failures,
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1791/1/testReport/org.apache.hadoop.hbase.client/TestFromClientSide5/precommit_checks___yetus_jdk11_hadoop3_checks___testScanMetrics_2_/
> Flaky test, TestFromClientSide5.testScanMetrics[2]
> --
>
> Key: HBASE-24429
> URL: https://issues.apache.org/jira/browse/HBASE-24429
> Project: HBase
> Issue Type: Bug
> Components: scan
>Affects Versions: master
>Reporter: Huaxiang Sun
>Assignee: Clara Xiong
>Priority: Major
>
> I see this test failure with master branch, assume this is a flakey test.
> {code:java}
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.client.TestFromClientSide5.testScanMetrics[2]Failing
> for the past 1 build (Since #2 )Took 4.3 sec.Error MessageDid not count the
> result bytes expected:<60> but was:<180>Stacktracejava.lang.AssertionError:
> Did not count the result bytes expected:<60> but was:<180>
> at
> org.apache.hadoop.hbase.client.TestFromClientSide5.testScanMetrics(TestFromClientSide5.java:1007)
> Standard Output2020-05-25 13:42:55,389 INFO [Time-limited test]
> hbase.ResourceChecker(151): before:
> client.TestFromClientSide5#testCacheOnWriteEvictOnClose[0] Thread=6,
> OpenFileDescriptor=220, MaxFileDescriptor=1048576, SystemLoadAverage=2329,
> ProcessCount=18, AvailableMemoryMB=16442
> 2020-05-25 13:42:56,056 DEBUG [Time-limited test]
> hbase.HBaseTestingUtility(333): Setting hbase.rootdir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0
> 2020-05-25 13:42:56,073 INFO [Time-limited test]
> hbase.HBaseTestingUtility(1047): Starting up minicluster with option:
> StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1,
> rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1,
> createRootDir=false, createWALDir=false}
> 2020-05-25 13:42:56,075 INFO [Time-limited test]
> hbase.HBaseZKTestingUtility(83): Created new mini-cluster data directory:
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/cluster_21ceb59f-5cc8-f531-e8c5-108c3a2d3cbb,
> deleteOnExit=true
> 2020-05-25 13:42:56,076 INFO [Time-limited test]
> hbase.HBaseTestingUtility(1061): STARTING DFS
> 2020-05-25 13:42:56,077 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting test.cache.data to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/test.cache.data
> in system properties and HBase conf
> 2020-05-25 13:42:56,079 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting hadoop.tmp.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/hadoop.tmp.dir
> in system properties and HBase conf
> 2020-05-25 13:42:56,080 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting hadoop.log.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/hadoop.log.dir
> in system properties and HBase conf
> 2020-05-25 13:42:56,082 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting mapreduce.cluster.local.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/mapreduce.cluster.local.dir
> in system properties and HBase conf
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24429) Flaky test, TestFromClientSide5.testScanMetrics[2]
[
https://issues.apache.org/jira/browse/HBASE-24429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huaxiang Sun reassigned HBASE-24429:
Assignee: Clara Xiong (was: Huaxiang Sun)
> Flaky test, TestFromClientSide5.testScanMetrics[2]
> --
>
> Key: HBASE-24429
> URL: https://issues.apache.org/jira/browse/HBASE-24429
> Project: HBase
> Issue Type: Bug
> Components: scan
>Affects Versions: master
>Reporter: Huaxiang Sun
>Assignee: Clara Xiong
>Priority: Major
>
> I see this test failure with master branch, assume this is a flakey test.
> {code:java}
> precommit checks / yetus jdk8 Hadoop3 checks /
> org.apache.hadoop.hbase.client.TestFromClientSide5.testScanMetrics[2]Failing
> for the past 1 build (Since #2 )Took 4.3 sec.Error MessageDid not count the
> result bytes expected:<60> but was:<180>Stacktracejava.lang.AssertionError:
> Did not count the result bytes expected:<60> but was:<180>
> at
> org.apache.hadoop.hbase.client.TestFromClientSide5.testScanMetrics(TestFromClientSide5.java:1007)
> Standard Output2020-05-25 13:42:55,389 INFO [Time-limited test]
> hbase.ResourceChecker(151): before:
> client.TestFromClientSide5#testCacheOnWriteEvictOnClose[0] Thread=6,
> OpenFileDescriptor=220, MaxFileDescriptor=1048576, SystemLoadAverage=2329,
> ProcessCount=18, AvailableMemoryMB=16442
> 2020-05-25 13:42:56,056 DEBUG [Time-limited test]
> hbase.HBaseTestingUtility(333): Setting hbase.rootdir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0
> 2020-05-25 13:42:56,073 INFO [Time-limited test]
> hbase.HBaseTestingUtility(1047): Starting up minicluster with option:
> StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1,
> rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1,
> createRootDir=false, createWALDir=false}
> 2020-05-25 13:42:56,075 INFO [Time-limited test]
> hbase.HBaseZKTestingUtility(83): Created new mini-cluster data directory:
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/cluster_21ceb59f-5cc8-f531-e8c5-108c3a2d3cbb,
> deleteOnExit=true
> 2020-05-25 13:42:56,076 INFO [Time-limited test]
> hbase.HBaseTestingUtility(1061): STARTING DFS
> 2020-05-25 13:42:56,077 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting test.cache.data to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/test.cache.data
> in system properties and HBase conf
> 2020-05-25 13:42:56,079 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting hadoop.tmp.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/hadoop.tmp.dir
> in system properties and HBase conf
> 2020-05-25 13:42:56,080 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting hadoop.log.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/hadoop.log.dir
> in system properties and HBase conf
> 2020-05-25 13:42:56,082 INFO [Time-limited test]
> hbase.HBaseTestingUtility(736): Setting mapreduce.cluster.local.dir to
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1770/yetus-jdk8-hadoop3-check/src/hbase-server/target/test-data/7fed70ca-373b-e224-a047-f4c0dd8789c0/mapreduce.cluster.local.dir
> in system properties and HBase conf
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24470) Add store file info when log meessages in HStore.
[
https://issues.apache.org/jira/browse/HBASE-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119984#comment-17119984
]
Hudson commented on HBASE-24470:
Results for branch branch-2.2
[build #880 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add store file info when log meessages in HStore.
> -
>
> Key: HBASE-24470
> URL: https://issues.apache.org/jira/browse/HBASE-24470
> Project: HBase
> Issue Type: Improvement
> Components: logging, regionserver
>Affects Versions: 3.0.0-alpha-1
>Reporter: song XinCun
>Assignee: song XinCun
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> Some log message do not have store file info when log, need to add it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24428) Priority compaction for recently split daughter regions
[
https://issues.apache.org/jira/browse/HBASE-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119983#comment-17119983
]
Hudson commented on HBASE-24428:
Results for branch branch-2.2
[build #880 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/880//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Priority compaction for recently split daughter regions
> ---
>
> Key: HBASE-24428
> URL: https://issues.apache.org/jira/browse/HBASE-24428
> Project: HBase
> Issue Type: Improvement
> Components: Compaction
>Reporter: Andrew Kyle Purtell
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.5
>
>
> We observe that under hotspotting conditions that splitting will proceed very
> slowly and the "_Cannot split region due to reference files being there_" log
> line will be logged excessively. (branch-1 based production.) This is because
> after a region is split it must be compacted before it can be split again.
> Reference files must be replaced by real HFiles, normal housekeeping
> performed during compaction. However if the regionserver is under excessive
> load, its compaction queues may become deep. The daughters of a recently
> split hotspotting region may themselves continue to hotspot and will rapidly
> need to split again. If the scheduled compaction work to remove/replace
> reference files is queued hundreds or thousands of compaction queue elements
> behind current, the recently split daughter regions will not be able to split
> again for a long time and may grow very large, producing additional
> complications (very large regions, very deep replication queues).
> To help avoid this condition we should prioritize the compaction of recently
> split daughter regions. Compaction requests include a {{priority}} field and
> CompactionRequest implements a comparator that sorts by this field. We
> already detect when a compaction request involves a region that has reference
> files, to ensure that it gets selected to be eligible for compaction, but we
> do not seem to prioritize the requests for post-split housekeeping. Split
> work should be placed at the top of the queue. Ensure that this is happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1807: HBASE-24343 Document how to configure the http request log
Apache-HBase commented on pull request #1807: URL: https://github.com/apache/hbase/pull/1807#issuecomment-636212353 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 5m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1807 | | Optional Tests | | | uname | Linux 3f1617006659 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06949ff6a6 | | Max. process+thread count | 51 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1807: HBASE-24343 Document how to configure the http request log
Apache-HBase commented on pull request #1807: URL: https://github.com/apache/hbase/pull/1807#issuecomment-636210447 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 1m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1807 | | Optional Tests | dupname asflicense | | uname | Linux 12dac46ba70a 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06949ff6a6 | | Max. process+thread count | 44 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1807: HBASE-24343 Document how to configure the http request log
Apache-HBase commented on pull request #1807: URL: https://github.com/apache/hbase/pull/1807#issuecomment-636210599 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 50s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1807 | | Optional Tests | | | uname | Linux cf4782aa6e95 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06949ff6a6 | | Max. process+thread count | 50 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1807/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1786: HBASE-24418 Consolidate Normalizer implementations
Apache-HBase commented on pull request #1786: URL: https://github.com/apache/hbase/pull/1786#issuecomment-636209250 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 24s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 44s | master passed | | +1 :green_heart: | checkstyle | 2m 42s | master passed | | +0 :ok: | refguide | 6m 33s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 16m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 43s | the patch passed | | +1 :green_heart: | checkstyle | 2m 40s | root: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +0 :ok: | refguide | 7m 14s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 14m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 16m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. | | | | 87m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1786 | | Optional Tests | dupname asflicense refguide xml spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux c8a923790ccb 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 06949ff6a6 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 123 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1786/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24343) Document how to configure the http request log
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24343 started by Nick Dimiduk. > Document how to configure the http request log > -- > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.3.0 >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1807: HBASE-24343 Document how to configure the http request log
ndimiduk opened a new pull request #1807: URL: https://github.com/apache/hbase/pull/1807 We don't have a section in the book on configuring logging, so I left it as comments in `conf/log4j.properties`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24343) Document how to configure the http request log
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24343: - Summary: Document how to configure the http request log (was: Document how to config request log for master and rs info server) > Document how to configure the http request log > -- > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.3.0 >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24473) Disable webui access log by default
[ https://issues.apache.org/jira/browse/HBASE-24473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24473. -- Resolution: Not A Problem Seems it's already disabled. I ran into it spamming my logs earlier... I wonder what I'd done. > Disable webui access log by default > --- > > Key: HBASE-24473 > URL: https://issues.apache.org/jira/browse/HBASE-24473 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > HBASE-24310 introduced an access log for http requests, which logs through > the standard logging facility. This adds a lot of noise to the master and > region server logs. Let's disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
Apache-HBase commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-636193000 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 9m 55s | branch-1 passed | | +1 :green_heart: | compile | 0m 40s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | compile | 0m 45s | branch-1 passed with JDK v1.7.0_262 | | +1 :green_heart: | checkstyle | 1m 51s | branch-1 passed | | +1 :green_heart: | shadedjars | 2m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 47s | branch-1 passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 41s | branch-1 passed with JDK v1.7.0_262 | | +0 :ok: | spotbugs | 3m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 59s | branch-1 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 53s | the patch passed | | +1 :green_heart: | compile | 0m 38s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javac | 0m 38s | the patch passed | | +1 :green_heart: | compile | 0m 45s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | javac | 0m 45s | the patch passed | | -1 :x: | checkstyle | 1m 40s | hbase-server: The patch generated 18 new + 245 unchanged - 36 fixed = 263 total (was 281) | | -1 :x: | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | shadedjars | 2m 47s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 4m 35s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 0m 31s | the patch passed with JDK v1.8.0_252 | | +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK v1.7.0_262 | | +1 :green_heart: | findbugs | 2m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 121m 25s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. | | | | 163m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1755 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 23c9fe6d5a60 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1755/out/precommit/personality/provided.sh | | git revision | branch-1 / 61fa21e | | Default Java | 1.7.0_262 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_252 /usr/lib/jvm/zulu-7-amd64:1.7.0_262 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/2/artifact/out/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/2/artifact/out/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/2/testReport/ | | Max. process+thread count | 4503 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1755/2/console | | versions | git=1.9.1 maven=3.0.5 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-22033) Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals
[ https://issues.apache.org/jira/browse/HBASE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22033: Fix Version/s: 2.4.0 3.0.0-alpha-1 Resolution: Fixed Status: Resolved (was: Patch Available) > Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals > -- > > Key: HBASE-22033 > URL: https://issues.apache.org/jira/browse/HBASE-22033 > Project: HBase > Issue Type: Task > Components: build, website >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > MJAVADOC-444 got into the 3.1.0 release of the maven-javadoc-plugin so now > there are versions of the aggregate javadoc goals that don't include a forked > build. > update our build to make use of this new feature. (a before/after on build > time would be nice to know as well) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24473) Disable webui access log by default
[ https://issues.apache.org/jira/browse/HBASE-24473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-24473: Assignee: Nick Dimiduk > Disable webui access log by default > --- > > Key: HBASE-24473 > URL: https://issues.apache.org/jira/browse/HBASE-24473 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > HBASE-24310 introduced an access log for http requests, which logs through > the standard logging facility. This adds a lot of noise to the master and > region server logs. Let's disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24473) Disable webui access log by default
Nick Dimiduk created HBASE-24473: Summary: Disable webui access log by default Key: HBASE-24473 URL: https://issues.apache.org/jira/browse/HBASE-24473 Project: HBase Issue Type: Sub-task Reporter: Nick Dimiduk HBASE-24310 introduced an access log for http requests, which logs through the standard logging facility. This adds a lot of noise to the master and region server logs. Let's disable it by default. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24343) Document how to config request log for master and rs info server
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24343: - Affects Version/s: 2.3.0 > Document how to config request log for master and rs info server > > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.3.0 >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24343) Document how to config request log for master and rs info server
[ https://issues.apache.org/jira/browse/HBASE-24343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reassigned HBASE-24343: Assignee: Nick Dimiduk > Document how to config request log for master and rs info server > > > Key: HBASE-24343 > URL: https://issues.apache.org/jira/browse/HBASE-24343 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24394) Add TTL to PE so that we can create a table with TTL
[ https://issues.apache.org/jira/browse/HBASE-24394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24394: - Fix Version/s: (was: 2.2.6) (was: 2.1.10) (was: 2.3.0) (was: 3.0.0-alpha-1) > Add TTL to PE so that we can create a table with TTL > > > Key: HBASE-24394 > URL: https://issues.apache.org/jira/browse/HBASE-24394 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > It may be very helpful to add TTL while creating the table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23958) Balancer keeps balancing indefinitely
[ https://issues.apache.org/jira/browse/HBASE-23958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23958: - Component/s: master Balancer > Balancer keeps balancing indefinitely > -- > > Key: HBASE-23958 > URL: https://issues.apache.org/jira/browse/HBASE-23958 > Project: HBase > Issue Type: Bug > Components: Balancer, master, Region Assignment >Affects Versions: 2.0.2 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > Before raising this issue - am not sure if this got fixed directly or > indirectly in other latest versions of hbase. > The steps are > 1) Create a cluster and create some tables. (assume we have RS 1,2,3, 4 and > 5) > 2) After the table creation and some ops done, the cluster was restarted. Due > to this some regions are in RIT. the RIT in progress was to be assigned to > RS 3. > 3) After the cluster comes back RS 3 and 4 are stopped. (RS 3 will have > newer timestamp) > 4) Now the master that comes up sees there are some RIT in place and tries to > load the entries to process the procedures again. As part of this the > RegionStateStore is populated with the old RS 3 hostname. (older timestamp). > This adds to the ServerStateNode creating a RS 3 with old timestamp as one > server. > 5) Now after the master restarts and all regions assigned, the balancer > infinitely tries to balance the region to the RS 3 (old timestamp server) > thinking it is part of the cluster. > 6)the other problem is the MoveProcedure has the target as RS 3 (with old > timestamp) but the AM realizes that it is a down server and move it to the > one of the active server. But this is not recorded anywhere. > I will continue to check the latest code if this case is valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23958) Balancer keeps balancing indefinitely
[ https://issues.apache.org/jira/browse/HBASE-23958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23958: - Fix Version/s: (was: 2.3.0) > Balancer keeps balancing indefinitely > -- > > Key: HBASE-23958 > URL: https://issues.apache.org/jira/browse/HBASE-23958 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Affects Versions: 2.0.2 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > > Before raising this issue - am not sure if this got fixed directly or > indirectly in other latest versions of hbase. > The steps are > 1) Create a cluster and create some tables. (assume we have RS 1,2,3, 4 and > 5) > 2) After the table creation and some ops done, the cluster was restarted. Due > to this some regions are in RIT. the RIT in progress was to be assigned to > RS 3. > 3) After the cluster comes back RS 3 and 4 are stopped. (RS 3 will have > newer timestamp) > 4) Now the master that comes up sees there are some RIT in place and tries to > load the entries to process the procedures again. As part of this the > RegionStateStore is populated with the old RS 3 hostname. (older timestamp). > This adds to the ServerStateNode creating a RS 3 with old timestamp as one > server. > 5) Now after the master restarts and all regions assigned, the balancer > infinitely tries to balance the region to the RS 3 (old timestamp server) > thinking it is part of the cluster. > 6)the other problem is the MoveProcedure has the target as RS 3 (with old > timestamp) but the AM realizes that it is a down server and move it to the > one of the active server. But this is not recorded anywhere. > I will continue to check the latest code if this case is valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24132) Upgrade to Apache ZooKeeper 3.5.7
[ https://issues.apache.org/jira/browse/HBASE-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24132. -- Resolution: Fixed > Upgrade to Apache ZooKeeper 3.5.7 > - > > Key: HBASE-24132 > URL: https://issues.apache.org/jira/browse/HBASE-24132 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > Apache ZooKeeper 3.5.7 has been released, HDFS and other projects have > updated the dependent zookeeper verison. Perhaps Hbase may update as well. > Some of the interfaces are changed in this zookeeper version . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24132) Upgrade to Apache ZooKeeper 3.5.7
[ https://issues.apache.org/jira/browse/HBASE-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24132: - Release Note: HBase ships ZooKeeper 3.5.x. Was the EOL'd 3.4.x. 3.5.x client can talk to 3.4.x ensemble. The ZooKeeper project has built a [FAQ](https://cwiki.apache.org/confluence/display/ZOOKEEPER/Upgrade+FAQ) that documents known issues and work-arounds when upgrading existing deployments. was:HBase ships ZooKeeper 3.5.x. Was the EOL'd 3.4.x. 3.5.x client can talk to 3.4.x ensemble. > Upgrade to Apache ZooKeeper 3.5.7 > - > > Key: HBASE-24132 > URL: https://issues.apache.org/jira/browse/HBASE-24132 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.2.3 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > Apache ZooKeeper 3.5.7 has been released, HDFS and other projects have > updated the dependent zookeeper verison. Perhaps Hbase may update as well. > Some of the interfaces are changed in this zookeeper version . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #1749: HBASE-24367 ScheduledChore log elapsed timespan in a human-friendly format
ndimiduk commented on pull request #1749: URL: https://github.com/apache/hbase/pull/1749#issuecomment-636180996 You have a new patch for us, master @apurtell ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-23639) Moving classes out of hbase-it /test for direct API use of chaos.
[ https://issues.apache.org/jira/browse/HBASE-23639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23639: - Fix Version/s: (was: 2.3.0) > Moving classes out of hbase-it /test for direct API use of chaos. > - > > Key: HBASE-23639 > URL: https://issues.apache.org/jira/browse/HBASE-23639 > Project: HBase > Issue Type: Improvement > Components: API, integration tests, test >Affects Versions: master >Reporter: Mihir Monani >Assignee: Lokesh Khurana >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > Attachments: HBASE-23639-master.patch > > > In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager > which has some part of implementation code for Chaos Action. > > If any user wants to create private implementation which resembles > HBaseClusterManager, then it has to be merged with hbase-it package as > ClusterManager Interface and it's extended implementations are private. > > We should make them public so anyone can have their own implementation and > need to be merged with hbase-it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24463) Allow operator to limit total normalization work per invocation
[ https://issues.apache.org/jira/browse/HBASE-24463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24463: - Component/s: Normalizer > Allow operator to limit total normalization work per invocation > --- > > Key: HBASE-24463 > URL: https://issues.apache.org/jira/browse/HBASE-24463 > Project: HBase > Issue Type: Improvement > Components: master, Normalizer >Reporter: Nick Dimiduk >Priority: Major > > During review on HBASE-24418, we observed that there's no way for an operator > to limit the total amount of work a normalizer invocation will do. One > suggestion was allowing settings for maximum number of plan executions per > table or maximum number of plan executions per run. I've seen other systems > limit the amount of CPU time permitted for a given run. > At least we have a run lock that prevents multiple invocations from running > concurrently, which should prevent one run from starting before the previous > one finished. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24465) Normalizer should consider region max file size when planning merges
[
https://issues.apache.org/jira/browse/HBASE-24465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24465:
-
Component/s: Normalizer
> Normalizer should consider region max file size when planning merges
>
>
> Key: HBASE-24465
> URL: https://issues.apache.org/jira/browse/HBASE-24465
> Project: HBase
> Issue Type: Improvement
> Components: master, Normalizer
>Reporter: Nick Dimiduk
>Priority: Major
>
> When the normalizer plans its actions, it does not consider
> {{hbase.hregion.max.filesize}}. This means it could get into a merge/split
> loop in collaboration with the region server: the normalizer sees two regions
> are smaller than the table average, it merges them. The resulting region is
> larger that this max file size, so the region server splits them. Repeat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23349) Low refCount preventing archival of compacted away files
[
https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-23349:
-
Fix Version/s: (was: 2.3.0)
2.4.0
> Low refCount preventing archival of compacted away files
>
>
> Key: HBASE-23349
> URL: https://issues.apache.org/jira/browse/HBASE-23349
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.6.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0
>
>
> We have observed that refCount on compacted away store files as low as 1 is
> prevent archival.
> {code:java}
> regionserver.HStore - Can't archive compacted file
> hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9
> because of either isCompactedAway=true or file has reference,
> isReferencedInReads=true, refCount=1, skipping for now.
> {code}
> We should come up with core code (run as part of discharger thread)
> gracefully resolve reader lock issue by resetting ongoing scanners to start
> pointing to new store files instead of compacted away store files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24464) Normalizer should have configuration for minimum region size
[
https://issues.apache.org/jira/browse/HBASE-24464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24464:
-
Component/s: Normalizer
> Normalizer should have configuration for minimum region size
>
>
> Key: HBASE-24464
> URL: https://issues.apache.org/jira/browse/HBASE-24464
> Project: HBase
> Issue Type: Improvement
> Components: master, Normalizer
>Reporter: Nick Dimiduk
>Priority: Major
>
> Another issue raised in PR review on HBASE-24418. With the current
> implementation, the lower-bound on how small the normalizer will make a
> region is its lowest granularity of size measurement: 1mb. That means it will
> happily balance a table out to be 1000 * 1mb regions (probably not likely,
> but seems plausible). The proposal was to add configuration knob that
> specifies the lower bound on region size as a guard against excessive
> splitting, something of an analogue to {{hbase.hregion.max.filesize}}.
> I'm not convinced this is really needed; I think the table would need to have
> been crafted intentionally to trigger this behavior. However, it seems like a
> reasonable guard rail to install. Discuss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24419) Normalizer merge plans should consider more than 2 regions when possible
[ https://issues.apache.org/jira/browse/HBASE-24419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24419: - Component/s: Normalizer > Normalizer merge plans should consider more than 2 regions when possible > > > Key: HBASE-24419 > URL: https://issues.apache.org/jira/browse/HBASE-24419 > Project: HBase > Issue Type: Improvement > Components: master, Normalizer >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Nick Dimiduk >Priority: Major > > The merge plans produced by the normalizer operate over two regions. Our > merge operation supports multiple regions in a single request. When there are > multiple merge plans generated over contiguous region space, these should be > collapsed into a single merge operation. This should automatically honor > whatever existing configuration settings exist limiting the number of regions > that can participate in a merge procedure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24418) Consolidate Normalizer implementations
[
https://issues.apache.org/jira/browse/HBASE-24418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-24418:
-
Component/s: Normalizer
> Consolidate Normalizer implementations
> --
>
> Key: HBASE-24418
> URL: https://issues.apache.org/jira/browse/HBASE-24418
> Project: HBase
> Issue Type: Task
> Components: master, Normalizer
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Blocker
> Fix For: 2.3.0
>
>
> After HBASE-22285, we have two implementations of {{RegionNormalizer}}, that
> have different feature sets and different configurations. I think these can
> be combined into a single implementation, with clear, decoupled configuration
> parameters. At least on branch-2.3, there's too many subsequent changes for
> HBASE-22285 to revert cleanly, so I'll use this ticket to consolidate the
> implementations.
> If you have issues with the current normalizer, speak up here and we can
> include them, or add them as sub-tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] busbey commented on pull request #1796: HBASE-22033 Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals
busbey commented on pull request #1796: URL: https://github.com/apache/hbase/pull/1796#issuecomment-636173544 thanks everyone! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] asfgit closed pull request #1796: HBASE-22033 Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals
asfgit closed pull request #1796: URL: https://github.com/apache/hbase/pull/1796 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-22033) Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals
[ https://issues.apache.org/jira/browse/HBASE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22033: Summary: Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals (was: Update to maven-javadoc-plugin 3.1.0 and switch to non-forking aggregate goals) > Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals > -- > > Key: HBASE-22033 > URL: https://issues.apache.org/jira/browse/HBASE-22033 > Project: HBase > Issue Type: Task > Components: build, website >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > MJAVADOC-444 got into the 3.1.0 release of the maven-javadoc-plugin so now > there are versions of the aggregate javadoc goals that don't include a forked > build. > update our build to make use of this new feature. (a before/after on build > time would be nice to know as well) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1793: HBASE-23941 : FilterBy operator support in get_slowlog_responses API
Apache-HBase commented on pull request #1793: URL: https://github.com/apache/hbase/pull/1793#issuecomment-636158344 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 52s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | checkstyle | 2m 24s | master passed | | +0 :ok: | refguide | 5m 26s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 17m 45s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 18s | the patch passed | | -0 :warning: | checkstyle | 2m 40s | root: The patch generated 2 new + 76 unchanged - 0 fixed = 78 total (was 76) | | -0 :warning: | rubocop | 0m 16s | The patch generated 7 new + 326 unchanged - 1 fixed = 333 total (was 327) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 6m 11s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 13m 18s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 7m 8s | the patch passed | | +1 :green_heart: | spotbugs | 19m 51s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. | | | | 96m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1793 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool rubocop refguide | | uname | Linux 1f2e423d6af6 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / bda2094ae5 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/branch-site/book.html | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | rubocop | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 124 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server hbase-shell . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1793/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 rubocop=0.80.0 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
Apache-HBase commented on pull request #1552: URL: https://github.com/apache/hbase/pull/1552#issuecomment-636139889 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 1m 21s | master passed | | +1 :green_heart: | shadedjars | 6m 16s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | master passed | | -0 :warning: | patch | 7m 28s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 0s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 38s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 227m 49s | hbase-server in the patch passed. | | | | 257m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1552 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 56c4f7447e93 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / da2e03bb3b | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/testReport/ | | Max. process+thread count | 2520 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
virajjasani commented on a change in pull request #1806:
URL: https://github.com/apache/hbase/pull/1806#discussion_r432677250
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
##
@@ -245,61 +238,56 @@ public ClusterId getClusterId() {
* directory with necessary bootup files).
*/
private Path checkRootDir(final Path rd, final Configuration c, final
FileSystem fs)
- throws IOException {
+throws IOException {
// If FS is in safe mode wait till out of it.
FSUtils.waitOnSafeMode(c, c.getInt(HConstants.THREAD_WAKE_FREQUENCY, 10 *
1000));
// Filesystem is good. Go ahead and check for hbase.rootdir.
+FileStatus status;
+try {
+ status = fs.getFileStatus(rd);
+} catch (FileNotFoundException e) {
+ status = null;
+}
try {
- if (!fs.exists(rd)) {
-fs.mkdirs(rd);
+ if (status == null) {
+if (!fs.mkdirs(rd)) {
+ throw new IOException("Can not create root dir: " + rd);
+}
// DFS leaves safe mode with 0 DNs when there are 0 blocks.
// We used to handle this by checking the current DN count and waiting
until
// it is nonzero. With security, the check for datanode count doesn't
work --
// it is a privileged op. So instead we adopt the strategy of the
jobtracker
// and simply retry file creation during bootstrap indefinitely. As
soon as
// there is one datanode it will succeed. Permission problems should
have
// already been caught by mkdirs above.
-FSUtils.setVersion(fs, rd, c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
- 10 * 1000), c.getInt(HConstants.VERSION_FILE_WRITE_ATTEMPTS,
-HConstants.DEFAULT_VERSION_FILE_WRITE_ATTEMPTS));
+FSUtils.setVersion(fs, rd, c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
10 * 1000), c.getInt(
+ HConstants.VERSION_FILE_WRITE_ATTEMPTS,
HConstants.DEFAULT_VERSION_FILE_WRITE_ATTEMPTS));
} else {
-if (!fs.isDirectory(rd)) {
- throw new IllegalArgumentException(rd.toString() + " is not a
directory");
+if (!status.isDirectory()) {
+ throw new IllegalArgumentException(rd + " is not a directory");
}
// as above
-FSUtils.checkVersion(fs, rd, true,
c.getInt(HConstants.THREAD_WAKE_FREQUENCY,
- 10 * 1000), c.getInt(HConstants.VERSION_FILE_WRITE_ATTEMPTS,
+FSUtils.checkVersion(fs, rd, true,
c.getInt(HConstants.THREAD_WAKE_FREQUENCY, 10 * 1000),
+ c.getInt(HConstants.VERSION_FILE_WRITE_ATTEMPTS,
HConstants.DEFAULT_VERSION_FILE_WRITE_ATTEMPTS));
}
} catch (DeserializationException de) {
- LOG.error(HBaseMarkers.FATAL, "Please fix invalid configuration for "
-+ HConstants.HBASE_DIR, de);
+ LOG.error(HBaseMarkers.FATAL, "Please fix invalid configuration for " +
HConstants.HBASE_DIR,
+de);
throw new IOException(de);
} catch (IllegalArgumentException iae) {
- LOG.error(HBaseMarkers.FATAL, "Please fix invalid configuration for "
-+ HConstants.HBASE_DIR + " " + rd.toString(), iae);
+ LOG.error(HBaseMarkers.FATAL,
+"Please fix invalid configuration for " + HConstants.HBASE_DIR + " " +
rd.toString(), iae);
Review comment:
Would you like to use placeholders `{}` for HConstants.HBASE_DIR and
rd.toString()? Also, only `rd` should suffice.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
##
@@ -245,61 +238,56 @@ public ClusterId getClusterId() {
* directory with necessary bootup files).
*/
private Path checkRootDir(final Path rd, final Configuration c, final
FileSystem fs)
Review comment:
The purpose of this method is to check if rootdir exists right? If it
can't get the Path obj, it throws Exception. If config for rootdir is not
valid, it throws Exception. However, after assigning `clusterId` of the object,
we don't really need to return path `rd` (hopefully, it's not getting mutated
internally) which was passed in the argument already. I believe this method
should be `private void checkRootDir()`. WDYT?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java
##
@@ -64,41 +72,68 @@ public TableOperationType getTableOperationType() {
return TableOperationType.CREATE;
}
+ private static void writeFsLayout(Path rootDir, Configuration conf) throws
IOException {
+LOG.info("BOOTSTRAP: creating hbase:meta region");
+FileSystem fs = rootDir.getFileSystem(conf);
+Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
+if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
+ LOG.warn("Can not delete partial created meta table, continue...");
+}
+// Bootstrapping, make sure blockcache is off. Else, one will be
+/
[GitHub] [hbase] Apache-HBase commented on pull request #1552: HBASE-24205 Create metric to know the number of reads that happens fr…
Apache-HBase commented on pull request #1552: URL: https://github.com/apache/hbase/pull/1552#issuecomment-636136273 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 18s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 50s | master passed | | +1 :green_heart: | compile | 1m 39s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 22s | hbase-hadoop-compat in master failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | | -0 :warning: | patch | 7m 59s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 46s | the patch passed | | +1 :green_heart: | compile | 1m 33s | the patch passed | | +1 :green_heart: | javac | 1m 33s | the patch passed | | +1 :green_heart: | shadedjars | 6m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 22s | hbase-hadoop-compat in the patch failed. | | -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 215m 9s | hbase-server in the patch failed. | | | | 248m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1552 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux df9b3d9a0807 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / da2e03bb3b | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/testReport/ | | Max. process+thread count | 3001 (vs. ulimit of 12500) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1552/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy
[ https://issues.apache.org/jira/browse/HBASE-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Sadineni updated HBASE-24472: -- Issue Type: Improvement (was: Test) > Enhanced version of KeyPrefixRegionSplitPolicy > -- > > Key: HBASE-24472 > URL: https://issues.apache.org/jira/browse/HBASE-24472 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Anil Sadineni >Priority: Major > > With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, > splitting regions policy is limited to either fixed length or based on > delimiter. With Yarn Application Timeline Server V2, as discussed in > YARN-10077, it will be nice to have enhanced version of > DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users > to define the ordinance of delimiter to consider. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sguggilam commented on pull request #1755: HBASE-24069 Provide an ExponentialBackOffPolicy sleep between failed …
sguggilam commented on pull request #1755: URL: https://github.com/apache/hbase/pull/1755#issuecomment-636119423 @apurtell @bharathv Incorporated the code review comments, please take a look Also the Assignment Manager V2 already have the retry with back off logic for failed open/close requests (https://github.com/apache/hbase/blob/branch-2/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java#L360) , so the change is not needed for branch-2 and above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-22041) [k8s] The crashed node exists in onlineServer forever, and if it holds the meta data, master will start up hang.
[
https://issues.apache.org/jira/browse/HBASE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119827#comment-17119827
]
Michael Stack commented on HBASE-22041:
---
bq. I was able to reproduce this issue with ttl=1 as well as ttl=0 (which I
guess means no caching).
Ouch.
Any evidence that java process ever notices the DNS update?
Next I think would be looking at throwing away the Connection after N retries.
We keep trying for ever waiting on an SCP. If can create a new Connection
and keep going w/o disrupting any other ongoing RPCs., that sounds like way to
go here. Thanks [~timoha]
> [k8s] The crashed node exists in onlineServer forever, and if it holds the
> meta data, master will start up hang.
>
>
> Key: HBASE-22041
> URL: https://issues.apache.org/jira/browse/HBASE-22041
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Priority: Critical
> Attachments: bug.zip, hbasemaster.log, normal.zip
>
>
> while master fresh boot, we crash (kill- 9) the RS who hold meta. we find
> that the master startup fails and print thounds of logs like:
> {code:java}
> 2019-03-13 01:09:54,896 WARN [RSProcedureDispatcher-pool4-t1]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to java.net.ConnectException: Call to
> hadoop14/172.16.1.131:16020 failed on connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> syscall:getsockopt(..) failed: Connection refused:
> hadoop14/172.16.1.131:16020, try=0, retrying...
> 2019-03-13 01:09:55,004 WARN [RSProcedureDispatcher-pool4-t2]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=1, retrying...
> 2019-03-13 01:09:55,114 WARN [RSProcedureDispatcher-pool4-t3]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=2, retrying...
> 2019-03-13 01:09:55,219 WARN [RSProcedureDispatcher-pool4-t4]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=3, retrying...
> 2019-03-13 01:09:55,324 WARN [RSProcedureDispatcher-pool4-t5]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=4, retrying...
> 2019-03-13 01:09:55,428 WARN [RSProcedureDispatcher-pool4-t6]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=5, retrying...
> 2019-03-13 01:09:55,533 WARN [RSProcedureDispatcher-pool4-t7]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=6, retrying...
> 2019-03-13 01:09:55,638 WARN [RSProcedureDispatcher-pool4-t8]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552410583724 failed due to
> org.apache.hadoop.hbase.ipc.FailedServerException: Call to
> hadoop14/172.16.1.131:16020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: hadoop14/172.16.1.131:16020, try=7, retrying...
> 2019-03-13 01:09:55,755 WARN [RSProcedureDispatcher-pool4-t9]
> procedure.RSProcedureDispatcher: request to server
> hadoop14,16020,1552
[GitHub] [hbase] Apache-HBase commented on pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache-HBase commented on pull request #1806: URL: https://github.com/apache/hbase/pull/1806#issuecomment-636104082 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | compile | 1m 43s | master passed | | +1 :green_heart: | shadedjars | 5m 55s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 49s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 22s | the patch passed | | +1 :green_heart: | compile | 1m 41s | the patch passed | | +1 :green_heart: | javac | 1m 41s | the patch passed | | +1 :green_heart: | shadedjars | 5m 39s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 135m 17s | hbase-server in the patch passed. | | | | 162m 59s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1806 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux 88722deb863a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / da2e03bb3b | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/testReport/ | | Max. process+thread count | 4430 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy
[ https://issues.apache.org/jira/browse/HBASE-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119796#comment-17119796 ] Anil Sadineni commented on HBASE-24472: --- Proposed solution - New DelimitedOrdinalKeyPrefixRegionSplitPolicy will have below parameters - DelimitedOrdinalKeyPrefixRegionSplitPolicy.delimiter - Delimiter used to identify the prefix DelimitedOrdinalKeyPrefixRegionSplitPolicy.delimiterOrdinal - occurrence number to which and prefix to be defined. If this looks ok, will upload patch for review. Appreciate your thoughts on this. > Enhanced version of KeyPrefixRegionSplitPolicy > -- > > Key: HBASE-24472 > URL: https://issues.apache.org/jira/browse/HBASE-24472 > Project: HBase > Issue Type: Test > Components: regionserver >Reporter: Anil Sadineni >Priority: Major > > With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, > splitting regions policy is limited to either fixed length or based on > delimiter. With Yarn Application Timeline Server V2, as discussed in > YARN-10077, it will be nice to have enhanced version of > DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users > to define the ordinance of delimiter to consider. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy
[ https://issues.apache.org/jira/browse/HBASE-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Sadineni updated HBASE-24472: -- Summary: Enhanced version of KeyPrefixRegionSplitPolicy (was: Enhanced version of KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy) > Enhanced version of KeyPrefixRegionSplitPolicy > -- > > Key: HBASE-24472 > URL: https://issues.apache.org/jira/browse/HBASE-24472 > Project: HBase > Issue Type: Test > Components: regionserver >Reporter: Anil Sadineni >Priority: Major > > With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, > splitting regions policy is limited to either fixed length or based on > delimiter. With Yarn Application Timeline Server V2, as discussed in > YARN-10077, it will be nice to have enhanced version of > DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users > to define the ordinance of delimiter to consider. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #1806: HBASE-24471 The way we bootstrap meta table is confusing
Apache-HBase commented on pull request #1806: URL: https://github.com/apache/hbase/pull/1806#issuecomment-636100105 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | +1 :green_heart: | compile | 1m 59s | master passed | | +1 :green_heart: | shadedjars | 5m 46s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 59s | the patch passed | | +1 :green_heart: | compile | 2m 0s | the patch passed | | +1 :green_heart: | javac | 2m 0s | the patch passed | | +1 :green_heart: | shadedjars | 5m 45s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 38s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 58s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 124m 19s | hbase-server in the patch passed. | | | | 154m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.10 Server=19.03.10 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1806 | | Optional Tests | unit javac javadoc shadedjars compile | | uname | Linux a8e7d3a83c68 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / da2e03bb3b | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/testReport/ | | Max. process+thread count | 4436 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1806/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24472) Enhanced version of KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy
Anil Sadineni created HBASE-24472: - Summary: Enhanced version of KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy Key: HBASE-24472 URL: https://issues.apache.org/jira/browse/HBASE-24472 Project: HBase Issue Type: Test Components: regionserver Reporter: Anil Sadineni With KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy, splitting regions policy is limited to either fixed length or based on delimiter. With Yarn Application Timeline Server V2, as discussed in YARN-10077, it will be nice to have enhanced version of DelimitedKeyPrefixRegionSplitPolicy that will give more flexibility to users to define the ordinance of delimiter to consider. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] anoopsjohn commented on a change in pull request #1803: [HBASE-24468]Add region info when log meessages in HStore.
anoopsjohn commented on a change in pull request #1803:
URL: https://github.com/apache/hbase/pull/1803#discussion_r432638555
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
##
@@ -276,7 +276,8 @@ protected HStore(final HRegion region, final
ColumnFamilyDescriptor family,
// used by ScanQueryMatcher
long timeToPurgeDeletes =
Math.max(conf.getLong("hbase.hstore.time.to.purge.deletes", 0), 0);
-LOG.trace("Time to purge deletes set to {}ms in store {}",
timeToPurgeDeletes, this);
+LOG.trace("Time to purge deletes set to {}ms in store {} in region {}",
timeToPurgeDeletes,
Review comment:
In all the places we have this pattern "in store {} in region {}"
Better we can change toString(). Include region and store name there. Then
in every log we can just refer to 'this'
: some format like this for toString?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (HBASE-22287) inifinite retries on failed server in RSProcedureDispatcher
[
https://issues.apache.org/jira/browse/HBASE-22287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-22287.
---
Fix Version/s: 2.3.0
3.0.0-alpha-1
Hadoop Flags: Reviewed
Release Note: Add backoff. Avoid retrying every 100ms.
Assignee: Michael Stack
Resolution: Fixed
Pushed to branch-2.3+. Thanks for review [~zhangduo]
> inifinite retries on failed server in RSProcedureDispatcher
> ---
>
> Key: HBASE-22287
> URL: https://issues.apache.org/jira/browse/HBASE-22287
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> We observed this recently on some cluster, I'm still investigating the root
> cause however seems like the retries should have special handling for this
> exception; and separately probably a cap on number of retries
> {noformat}
> 2019-04-20 04:24:27,093 WARN [RSProcedureDispatcher-pool4-t1285]
> procedure.RSProcedureDispatcher: request to server ,17020,1555742560432
> failed due to java.io.IOException: Call to :17020 failed on local exception:
> org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the
> failed servers list: :17020, try=26603, retrying...
> {noformat}
> The corresponding worker is stuck
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24445) Improve default thread pool size for opening store files
[ https://issues.apache.org/jira/browse/HBASE-24445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119769#comment-17119769 ] Andrew Kyle Purtell commented on HBASE-24445: - HBASE-24436 proposes an alternative, waiting for a resolution there. > Improve default thread pool size for opening store files > > > Key: HBASE-24445 > URL: https://issues.apache.org/jira/browse/HBASE-24445 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > For each store open we create a CompletionService and also create a thread > pool for opening and closing store files. See HStore#openStoreFiles and > HRegion#getStoreFileOpenAndCloseThreadPool. By default this pool has only one > thread. It can be increased with "hbase.hstore.open.and.close.threads.max" > but this config value is then divided by number of stores in the region. > "hbase.hstore.open.and.close.threads.max" is also used to size other thread > pools for opening and closing the stores themselves, so it's an unfortunate > overloading. > We should have a configuration parameter that directly and simply tunes the > thread pool size for opening store files. Introduce a new configuration > parameter: "hbase.hstore.hfile.open.threads.max" which will define the upper > bound for a thread pool shared by the entire store for opening hfiles. The > default should be 1 to preserve default behavior. > Once this is done, we could increase this to 2, 4, 8, or more for increased > parallelism when opening store files without impact on other activities. The > time required to open all storefiles often dominates the total time for > bringing a region online. The thread pool will be shut down and eligible for > garbage collection once all files are loaded and the store is online. > Number of open threads should scale with the number of stores, so allocating > the pool at the store level continues to make sense. > Longer term we might try recursively decomposing the region open task with a > fork-join pool such that the opening of store files can be dynamically > parallelized in a probably superior way (conjecture pending a real attempt > with metrics) . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #1800: HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher
saintstack merged pull request #1800: URL: https://github.com/apache/hbase/pull/1800 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
