[jira] [Updated] (HBASE-24717) regions opened in read only mode failed when replaying the edits due to permission denied WRITE for regionDir
[
https://issues.apache.org/jira/browse/HBASE-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-24717:
-
Description:
Open region in read only mode, [HbaseTableSnapshotInputFormat], if RegionDir
has
recovered edits files under, (filesUnderRootDir), when read-only region read
the content, it will try to delete the content of recovered edits, including
filesUnderRootDir part
{code:java}
// code placeholder
for (Path file : filesUnderRootDir) {
if (!rootFS.delete(file, false)) {
LOG.error("Failed delete of {}", file);
} else {
LOG.debug("Deleted recovered.edits file={}", file);
}
}
{code}
however, read-only mode usually no WRITE access to delete it.
{code:java}
// code placeholder
org.apache.hadoop.security.AccessControlException: Permission denied:
user=, access=WRITE,
inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
{code}
it will throw excption and failed
was:
when open region in read only mode, [HbaseTableSnapshotInputFormat], if
RegionDir has
recovered edits files under, (filesUnderRootDir), when read-only region read
the content, it will try to delete the content of recovered edits, including
filesUnderRootDir part
{code:java}
// code placeholder
for (Path file : filesUnderRootDir) {
if (!rootFS.delete(file, false)) {
LOG.error("Failed delete of {}", file);
} else {
LOG.debug("Deleted recovered.edits file={}", file);
}
}
{code}
however, read-only mode usually no WRITE access to delete it.
{code:java}
// code placeholder
org.apache.hadoop.security.AccessControlException: Permission denied:
user=, access=WRITE,
inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
{code}
it will throw excption and failed
> regions opened in read only mode failed when replaying the edits due to
> permission denied WRITE for regionDir
> -
>
> Key: HBASE-24717
> URL: https://issues.apache.org/jira/browse/HBASE-24717
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: huan
>Assignee: huan
>Priority: Blocker
>
> Open region in read only mode, [HbaseTableSnapshotInputFormat], if RegionDir
> has
> recovered edits files under, (filesUnderRootDir), when read-only region read
> the content, it will try to delete the content of recovered edits, including
> filesUnderRootDir part
> {code:java}
> // code placeholder
> for (Path file : filesUnderRootDir) {
> if (!rootFS.delete(file, false)) {
> LOG.error("Failed delete of {}", file);
> } else {
> LOG.debug("Deleted recovered.edits file={}", file);
> }
> }
> {code}
> however, read-only mode usually no WRITE access to delete it.
> {code:java}
> // code placeholder
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=, access=WRITE,
> inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
> {code}
> it will throw excption and failed
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24717) regions opened in read only mode failed when replaying the edits due to permission denied WRITE for regionDir
[
https://issues.apache.org/jira/browse/HBASE-24717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huan updated HBASE-24717:
-
Summary: regions opened in read only mode failed when replaying the edits
due to permission denied WRITE for regionDir (was: regions opened in read only
mode failed when replay the edits due to permission denied WRITE for regionDir)
> regions opened in read only mode failed when replaying the edits due to
> permission denied WRITE for regionDir
> -
>
> Key: HBASE-24717
> URL: https://issues.apache.org/jira/browse/HBASE-24717
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: huan
>Assignee: huan
>Priority: Blocker
>
> when open region in read only mode, [HbaseTableSnapshotInputFormat], if
> RegionDir has
> recovered edits files under, (filesUnderRootDir), when read-only region read
> the content, it will try to delete the content of recovered edits, including
> filesUnderRootDir part
> {code:java}
> // code placeholder
> for (Path file : filesUnderRootDir) {
> if (!rootFS.delete(file, false)) {
> LOG.error("Failed delete of {}", file);
> } else {
> LOG.debug("Deleted recovered.edits file={}", file);
> }
> }
> {code}
> however, read-only mode usually no WRITE access to delete it.
> {code:java}
> // code placeholder
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=, access=WRITE,
> inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
> {code}
> it will throw excption and failed
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24717) regions opened in read only mode failed when replay the edits due to permission denied WRITE for regionDir
huan created HBASE-24717:
Summary: regions opened in read only mode failed when replay the
edits due to permission denied WRITE for regionDir
Key: HBASE-24717
URL: https://issues.apache.org/jira/browse/HBASE-24717
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: huan
Assignee: huan
when open region in read only mode, [HbaseTableSnapshotInputFormat], if
RegionDir has
recovered edits files under, (filesUnderRootDir), when read-only region read
the content, it will try to delete the content of recovered edits, including
filesUnderRootDir part
{code:java}
// code placeholder
for (Path file : filesUnderRootDir) {
if (!rootFS.delete(file, false)) {
LOG.error("Failed delete of {}", file);
} else {
LOG.debug("Deleted recovered.edits file={}", file);
}
}
{code}
however, read-only mode usually no WRITE access to delete it.
{code:java}
// code placeholder
org.apache.hadoop.security.AccessControlException: Permission denied:
user=, access=WRITE,
inode="/hbase/data/default/TABLENAME/d6777e449ed4d137b6a2f735653a7c3a/recovered.edits":hbase:hbase:drwxr-xr-x
{code}
it will throw excption and failed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24586) Add table level locality in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-24586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155886#comment-17155886 ] Zheng Wang commented on HBASE-24586: Pushed a new PR to branch2 yet. [~vjasani] > Add table level locality in table.jsp > - > > Key: HBASE-24586 > URL: https://issues.apache.org/jira/browse/HBASE-24586 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: HBASE-24586-afterpatch.png, HBASE-24586-beforepatch.png > > > For now, we only show locality for each region, it should be apply to table > as what Base Stats and Compactions does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] bsglz opened a new pull request #2048: Add table level locality in table.jsp
bsglz opened a new pull request #2048: URL: https://github.com/apache/hbase/pull/2048 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24404) Support flush a single column family of region
[ https://issues.apache.org/jira/browse/HBASE-24404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155842#comment-17155842 ] Zheng Wang commented on HBASE-24404: No prob, will do it later, thanks. [~wchevreuil] > Support flush a single column family of region > -- > > Key: HBASE-24404 > URL: https://issues.apache.org/jira/browse/HBASE-24404 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 3.0.0-alpha-1 >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > It is useful when we set many column families for a table, could avoid > generate too many small files. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] nyl3532016 commented on a change in pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
nyl3532016 commented on a change in pull request #2017:
URL: https://github.com/apache/hbase/pull/2017#discussion_r453145383
##
File path:
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
##
@@ -1842,7 +1842,7 @@ private void
countDownChildren(RootProcedureState procStack,
store.update(parent);
scheduler.addFront(parent);
LOG.info("Finished subprocedure pid={}, resume processing parent {}",
- procedure.getProcId(), parent);
+ procedure.getProcId(), parent.toString().replace("pid=","ppid="));
Review comment:
Yes, I have modified it like this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
Apache-HBase commented on pull request #2017: URL: https://github.com/apache/hbase/pull/2017#issuecomment-656970201 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 0s | branch-2 passed | | +1 :green_heart: | checkstyle | 0m 15s | branch-2 passed | | +1 :green_heart: | spotbugs | 0m 32s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | checkstyle | 0m 13s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 0m 40s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 12s | The patch does not generate ASF License warnings. | | | | 36m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2017 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ea12a296a29f 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ff2951e672 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-procedure U: hbase-procedure | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
Apache-HBase commented on pull request #2017: URL: https://github.com/apache/hbase/pull/2017#issuecomment-656969299 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 48s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 28s | branch-2 passed | | +1 :green_heart: | compile | 0m 22s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 0s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-procedure in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 0m 21s | the patch passed | | +1 :green_heart: | javac | 0m 21s | the patch passed | | +1 :green_heart: | shadedjars | 6m 3s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-procedure in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 44s | hbase-procedure in the patch passed. | | | | 29m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2017 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 80a179694483 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ff2951e672 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-procedure.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-procedure.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/testReport/ | | Max. process+thread count | 297 (vs. ulimit of 12500) | | modules | C: hbase-procedure U: hbase-procedure | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
Apache-HBase commented on pull request #2017: URL: https://github.com/apache/hbase/pull/2017#issuecomment-656968772 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 49s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | branch-2 passed | | +1 :green_heart: | compile | 0m 18s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 0m 19s | the patch passed | | +1 :green_heart: | javac | 0m 19s | the patch passed | | +1 :green_heart: | shadedjars | 5m 3s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 17s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 50s | hbase-procedure in the patch passed. | | | | 26m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2017 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a27c4d229d31 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ff2951e672 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/testReport/ | | Max. process+thread count | 373 (vs. ulimit of 12500) | | modules | C: hbase-procedure U: hbase-procedure | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2017/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2047: [HBASE-24714] Error message is displayed in the UI of table's compact…
Apache-HBase commented on pull request #2047: URL: https://github.com/apache/hbase/pull/2047#issuecomment-656955806 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 15s | master passed | | +1 :green_heart: | compile | 1m 50s | master passed | | +1 :green_heart: | shadedjars | 6m 57s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 33s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 54s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 27s | the patch passed | | +1 :green_heart: | compile | 2m 22s | the patch passed | | +1 :green_heart: | javac | 2m 22s | the patch passed | | +1 :green_heart: | shadedjars | 7m 30s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 30s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 48s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 39s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 188m 12s | hbase-server in the patch passed. | | | | 226m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2047 | | JIRA Issue | HBASE-24714 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6f28776dd765 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 724f0478ed | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/testReport/ | | Max. process+thread count | 3130 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155806#comment-17155806
]
Duo Zhang commented on HBASE-11288:
---
{quote}
Cache tier implementation is not part of this discussion unless you think it
needs to be. That the API locating ROOT content is simple and so easy to back
w/ a general cache is sufficient to our purposes here? Or do you think we need
to say more on this topic? (As to the cache implementation, I'd think we'd want
to avoid involving Masters in any caching solution if ROOT is located elsewhere
– involving Masters would undo a 'pro' of the 'general root table' direction as
you point out subsequently in your comment).
{quote}
I think this is a important now as [~toffer] keep saying that it should be the
same as 'master local region' but obviously, your argument later that we should
not involve masters means it is a different solition. Please give a clear
design. I was attacked at the first place because I did not give out a detailed
design, it hurts my feelings because it seems like a double standard.
So let's go back to the plan above? [~stack] please give a clear design on how
to implement the cache server for 'general root table' which do not need to
involve masters, and [~toffer] please go with your ITBLL on the POC to see if
we can get a useful result, and I will implement HBASE-24459 to show how to
deal with the caching for 'master local region'.
Thanks.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2047: [HBASE-24714] Error message is displayed in the UI of table's compact…
Apache-HBase commented on pull request #2047: URL: https://github.com/apache/hbase/pull/2047#issuecomment-656947208 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 17s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 31s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 6m 52s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 6s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 17s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 144m 0s | hbase-server in the patch passed. | | | | 177m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2047 | | JIRA Issue | HBASE-24714 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux bec908523e63 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 724f0478ed | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/testReport/ | | Max. process+thread count | 3916 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155801#comment-17155801 ] Duo Zhang commented on HBASE-11288: --- Generally speaking, caching is for solving hot root region, as a side effect, it could reduce the problem of master on critical path. And please stop saying things which do not exist yet. For addressing hot spot region, there are mainly two ways in HBase, one is split the hot region, the other is read replicas. The former one is not suitable for root region as it is not splittable, the latter one is not stable enough so we do not consider it. If you think you can fix them as a whole, please give a clear solution. Hot spot is a problem when HBase was born more than 10 years ago, and we still do not have a perfect solution to solve it, and even this issue is partly for solving the hot spot on meta region. You can not simply throw out a statement that ‘we should address them as a whole’ but do not give a solution, irresponsible. Thanks. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155797#comment-17155797 ] Francis Christopher Liu commented on HBASE-11288: - I think we are conflating the purpose of caching and that makes the evaluation a bit confusing. Here's my attempt at fleshing them out. And drawing some conclusions. There's essentially two cases: # Caching to prevent master to become part of read/write path - this use case is only presenet in "local master region". Caching here is used to reduce the effect of disadvantage. Bad enough to be required as part of any serious deployment(?). # Caching to avoid hot regions. - this is something potentially optional in "general root table". But arguably recommended in "local master region" given the sharing of master and root responsibilities in a single process. There is some nuance here as to why this caching use case is more important/recommended for "local master region" because of being colocated with the master especially when it comes to scaling(?). For 2, I think we are special casing root a little too much. For example in "general root table" implementation although splitting meta can reduce hot regions and help with scaling it does not eliminate it. In fact in our experience we run into hot meta regions more often than hot Root region. In which case I believe if we are considering caching as a solution for hot root region we need to think more holistically and think about a solution for Catalog Tables as a whole. Along those lines, although it's currently not a big deal for us, would it also make sense to consider availability for Catalog Tables as whole, which also can potentially be addressed with caching? The same can be said with the noisy neighbors argument does it really make sense to special case root instead of coming up with a holistic solution for Catalog Tables as a whole? It seems one potential advantage for "general root table" is being able to address issues with the Catalog Tables holistically potentially having a simpler and more elegant solution. The devil is in the details here of course and so the general question is how important is special casing root for any particular use case vs fixing it for catalog table as a whole. It at least seemed to me based on discussions and the design doc that the 2-tier was potentially that use case. What do you guys think? > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24716) Do the error handling for replication admin failures
[
https://issues.apache.org/jira/browse/HBASE-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal updated HBASE-24716:
Description:
[listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
Snippet:
{code:java}
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
{code}
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
was:
[listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
> Do the error handling for replication admin failures
>
>
> Key: HBASE-24716
> URL: https://issues.apache.org/jira/browse/HBASE-24716
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
>
> [listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
> for getting the list of peers along with their configuration is not a
> reliable API.
> It is not very robust to errors, logs FATAL and swallows the
> [exceptions|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
>
> Snippet:
> {code:java}
> catch (KeeperException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> } catch (ReplicationException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> }
> return peers;
> {code}
>
> The abortable (connection in
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155781#comment-17155781
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #15 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/15/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22738) Fallback to default group to choose RS when there are no RS in current group
[
https://issues.apache.org/jira/browse/HBASE-22738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155770#comment-17155770
]
Hudson commented on HBASE-22738:
Results for branch branch-2
[build #2741 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fallback to default group to choose RS when there are no RS in current group
>
>
> Key: HBASE-22738
> URL: https://issues.apache.org/jira/browse/HBASE-22738
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Reporter: Guanghao Zhang
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> We configure one regionserver for hbase system table. But when rolling
> upgrade, you need move the region to other regionservers. But because there
> are no other regionservers in this group, you cannot move the region...
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24663) Add procedure process time statistics UI
[
https://issues.apache.org/jira/browse/HBASE-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155772#comment-17155772
]
Hudson commented on HBASE-24663:
Results for branch branch-2
[build #2741 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add procedure process time statistics UI
>
>
> Key: HBASE-24663
> URL: https://issues.apache.org/jira/browse/HBASE-24663
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Junhong Xu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: screenshot-1.png
>
>
> Added in "Procedures & Locks" jsp.
> For the first version UI, we care about the process time of
> ServerCrashProcedure, TRSP, OpenRegionProcedure and CloseRegionProcedure.
> Plan to show the avg/P50/P90/min/max process time of these procedures.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24653) Show snapshot owner on Master WebUI
[
https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155771#comment-17155771
]
Hudson commented on HBASE-24653:
Results for branch branch-2
[build #2741 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2741/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Show snapshot owner on Master WebUI
> ---
>
> Key: HBASE-24653
> URL: https://issues.apache.org/jira/browse/HBASE-24653
> Project: HBase
> Issue Type: Improvement
>Reporter: Yi Mei
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: snapshot1.png, snapshot2.png
>
>
> Now Master UI shows lots of snapshot informations, and owner is also useful
> to find out who create this snapshot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2047: [HBASE-24714] Error message is displayed in the UI of table's compact…
Apache-HBase commented on pull request #2047: URL: https://github.com/apache/hbase/pull/2047#issuecomment-656910711 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 51s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 12s | master passed | | +1 :green_heart: | checkstyle | 1m 46s | master passed | | +1 :green_heart: | spotbugs | 3m 22s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | checkstyle | 1m 35s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 12s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 39m 46s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2047 | | JIRA Issue | HBASE-24714 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 711fac18d86f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 724f0478ed | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2047/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-8897) Add Suspend and Resume to Chaos Monkey
[
https://issues.apache.org/jira/browse/HBASE-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-8897.
-
Resolution: Not A Problem
Looks like this as since been implemented, via {{SIGSTOP}}, {{SIGCONT}}, in
{{HBaseClusterManager}}.
> Add Suspend and Resume to Chaos Monkey
> --
>
> Key: HBASE-8897
> URL: https://issues.apache.org/jira/browse/HBASE-8897
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Elliott Neil Clark
>Priority: Major
> Attachments: HBASE-8897-0.patch
>
>
> We should add suspend and resume to chaos monkey. They are already exposed
> through HBaseClusterManager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] joshelser commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
joshelser commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-656905422 I just had a chat with Surbhi and we walked through the code, end to end, talking about her solution here. tl;dr I'm with her on the solution, barring one exception where we remove the serialized `SpaceQuotaSnapshot` (column `u:p`) for a table which had both a namespace quota (now removed) and a table quota. Removing the `u:p` record when the table quota still applies would result in a RegionServer potentially move a table out of quota violation when it should still remain in violation. A sketch of how space quotas work again, because I had to remind myself: 1. RegionServers report Region space utilization to the master 2. Master holds this in memory, eventually computing table-level and namespace-level utilization. 3. Master computes whether each table is or is not in violation, and persists that (with usage and limit) into a SpaceQuotaSnapshot (`u:p` column) for each table. 4. RegionServers will read these `SpaceQuotaSnapshots` from `hbase:quota` and use this to determine whether or not to reject the new Mutations in `multi()`. That is, when we drop a quota for a namespace, we should clean up all SpaceQuotaSnapshot records in hbase:quota that do not otherwise have a table-level space quota defined on them. for example, ``` hbase(main):015:0> scan 'hbase:quota' ROW COLUMN+CELL n.test column=q:s, timestamp=1591306396362, value=PBUF\x1A\x07\x08.. t.test:table1 column=q:s, timestamp=1591274418036, value=PBUF\x1A\x07\x08.. t.test:table1 column=u:p, timestamp=1591306447760, value=\x0A\x04\x08.. t.test:table2 column=u:p, timestamp=1591306447758, value=\x0A\x04\x08.. ``` when the space quota on namespace `test` is deleted, we would want to delete `t.test:table2 u:p`, but we would not want to delete `t.test:table1 u:p` because there is a quota defined on this table (`t.test:table1 q:s`). Normally, `QuotaObserverChore` (in the master) is pushing new `SpaceQuotaSnapshot` records into `hbase:quota`, but this class doesn't handle the cleanup. The method that Surbhi has modified is what MasterQuotaManager is ultimately calling (after a couple of levels of indirection), so making this change inside `QuotaUtil` appears to be the right thing to me, hiding the complexity of what it means to delete a namespace quota :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24714) Error message is displayed in the UI of table's compaction state if any region of that table is not open.
[ https://issues.apache.org/jira/browse/HBASE-24714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155726#comment-17155726 ] Sanjeet Nishad commented on HBASE-24714: PR raised for this issue: https://github.com/apache/hbase/pull/2047 > Error message is displayed in the UI of table's compaction state if any > region of that table is not open. > - > > Key: HBASE-24714 > URL: https://issues.apache.org/jira/browse/HBASE-24714 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: master, 2.2.3 >Reporter: Sanjeet Nishad >Assignee: Sanjeet Nishad >Priority: Major > Attachments: ErrorMessageinUI.png > > > When a region of a table is in states closed, merging, splitting, etc, the > table's compaction state displays error stack trace. > > !ErrorMessageinUI.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] sanjeetnishad95 opened a new pull request #2047: [HBASE-24714] Error message is displayed in the UI of table's compact…
sanjeetnishad95 opened a new pull request #2047: URL: https://github.com/apache/hbase/pull/2047 …ion state if any region of that table is not open. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155724#comment-17155724
]
Bharath Vissapragada commented on HBASE-24603:
--
Committed to branch-2.1 and 2.2. Let me know if you'd like it in any other
branches too.
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.1.10, 2.2.6
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24603) Zookeeper sync() call is async
[
https://issues.apache.org/jira/browse/HBASE-24603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-24603:
-
Fix Version/s: 2.2.6
2.1.10
> Zookeeper sync() call is async
> --
>
> Key: HBASE-24603
> URL: https://issues.apache.org/jira/browse/HBASE-24603
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.1.10, 2.2.6
>
>
> Here is the method that does a sync() of lagging followers with leader in the
> quorum. We rely on this to see a consistent snapshot of ZK data from multiple
> clients. However the problem is that the underlying sync() call is actually
> asynchronous since we are passing a 'null' call back. See the ZK API
> [doc|https://zookeeper.apache.org/doc/r3.5.7/apidocs/zookeeper-server/index.html]
> for details. The end-result is that sync() doesn't guarantee that it has
> happened by the time it returns.
> {noformat}
> /**
>* Forces a synchronization of this ZooKeeper client connection.
>*
>* Executing this method before running other methods will ensure that the
>* subsequent operations are up-to-date and consistent as of the time that
>* the sync is complete.
>*
>* This is used for compareAndSwap type operations where we need to read the
>* data of an existing node and delete or transition that node, utilizing
> the
>* previously read version and data. We want to ensure that the version
> read
>* is up-to-date from when we begin the operation.
>*/
> public void sync(String path) throws KeeperException {
> this.recoverableZooKeeper.sync(path, null, null);
> }
> {noformat}
> We rely on this heavily (at least in the older branches that do ZK based
> region assignment). In branch-1 we saw weird "BadVersionException" exceptions
> in RITs because of the inconsistent view of the ZK snapshot. It could
> manifest differently in other branches. Either way, this is something we need
> to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24716) Do the error handling for replication admin failures
[
https://issues.apache.org/jira/browse/HBASE-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal updated HBASE-24716:
Description:
[listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
was:
[listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
> Do the error handling for replication admin failures
>
>
> Key: HBASE-24716
> URL: https://issues.apache.org/jira/browse/HBASE-24716
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
>
> [listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
> for getting the list of peers along with their configuration is not a
> reliable API.
> It is not very robust to errors, logs FATAL and swallows the
> [exceptions|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
>
> Snippet:
> catch (KeeperException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> } catch (ReplicationException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> }
> return peers;
>
> The abortable (connection in this case) also doesn't abort the region
[jira] [Updated] (HBASE-24716) Do the error handling for replication admin failures
[
https://issues.apache.org/jira/browse/HBASE-24716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandeep Pal updated HBASE-24716:
Description:
[listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
was:
[listPeerConfigs()|[https://git.soma.salesforce.com/bigdata-packaging/hbase/blob/1.6.0-sfdc-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
> Do the error handling for replication admin failures
>
>
> Key: HBASE-24716
> URL: https://issues.apache.org/jira/browse/HBASE-24716
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Sandeep Pal
>Assignee: Sandeep Pal
>Priority: Major
>
> [listPeerConfigs()|https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]
> for getting the list of peers along with their configuration is not a
> reliable API.
> It is not very robust to errors, logs FATAL and swallows the
> [exceptions|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]
>
> Snippet:
> catch (KeeperException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> } catch (ReplicationException e) {
> this.abortable.abort("Cannot get the list of peers ", e);
> }
> return peers;
>
> The abortable (connection in this case)
[GitHub] [hbase] skochhar commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
skochhar commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-656875501 > > My change removes these QUOTA_FAMILY_USAGE column family for the tables when a namespace quota is deleted and allows the user to insert into the tables in the namespace when the violated space quota is removed. > > > But I thought you should fix the problem not in QuotaUtil class, maybe in MasterQuotaManager? Keep QuotaUtil simple and fix this in upper layer. > > Thanks for the explanation in [#1935 (review)](https://github.com/apache/hbase/pull/1935#pullrequestreview-439900394), Surbhi. This helps tremendously. > > I'm not sure that removing the `u:p` cell is the correct solution. These columns are just tracking the _size_ of the table. In themselves, they aren't controlling anything (if memory serves). Instead, we should know that when we delete the NamespaceQuota, MasterQuotaManager needs to become aware of this (somehow) and remove any SpaceQuotaViolationPolicies that are in effect. > > I suspect your fix does work, but for the wrong reason. If you remove the sizes for these tables, the Master will simply think that the table is no longer in validation. If you have a table-level quota also defined (like I commented in-code), I think this change will incorrectly cause that table-level to not be applied. Thanks Josh, I tested that my fix does not remove any table-level quotas. If a namespace "test" has 2 tables "test:table1" and "test:table2". And namespace "test" has a space quota on it and the "test.table1" also has a table-quota on it so you see 4 rows in quota table, the row for the namespace quota, the row for the table quota and the usage rows for the tables under the namespace. Showing you below scan of hbase:quota table with a namespace "test" and table "test:table1" that has the space quota hbase(main):015:0> scan 'hbase:quota' ROW COLUMN+CELL n.test column=q:s, timestamp=1591306396362, value=PBUF\x1A\x07\x08\.. t.test:table1 column=q:s, timestamp=1591274418036, value=PBUF\x1A\x07\x08\.. t.test:table1 column=u:p, timestamp=1591306447760, value=\x0A\x04\x08\.. t.test:table2 column=u:p, timestamp=1591306447758, value=\x0A\x04\x08.. hbase(main):018:0> list_quotas OWNERQUOTAS NAMESPACE => test TYPE => SPACE, NAMESPACE => test, LIMIT => 2.00M, VIOLATION_POLICY => NO_INSERTS TABLE => test:table1TYPE => SPACE, TABLE => test:table1, LIMIT => 30.00M, VIOLATION_POLICY => NO_INSERTS 2 row(s) My fix only removes the column family QUOTA_FAMILY_USAGE(u:p here) for the table and the QUOTA_FAMILY_INFO(q:s here) stay with my fix, so the table-quotas still get applied once the namespace quota is removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24716) Do the error handling for replication admin failures
Sandeep Pal created HBASE-24716:
---
Summary: Do the error handling for replication admin failures
Key: HBASE-24716
URL: https://issues.apache.org/jira/browse/HBASE-24716
Project: HBase
Issue Type: Improvement
Components: Replication
Reporter: Sandeep Pal
Assignee: Sandeep Pal
[listPeerConfigs()|[https://git.soma.salesforce.com/bigdata-packaging/hbase/blob/1.6.0-sfdc-1/hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java#L295]]
for getting the list of peers along with their configuration is not a reliable
API.
It is not very robust to errors, logs FATAL and swallows the
[exceptions|[https://github.com/apache/hbase/blob/branch-1/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java#L254]]
Snippet:
catch (KeeperException e) {
this.abortable.abort("Cannot get the list of peers ", e);
} catch (ReplicationException e) {
this.abortable.abort("Cannot get the list of peers ", e);
}
return peers;
The abortable (connection in this case) also doesn't abort the region server
and just logs. This makes upstream believe that there is nothing wrong and
proceed without any action which is not good.
{code:java}
2020-07-07 23:11:37,857 FATAL [14774961,peer_id]
client.ConnectionManager$HConnectionImplementation - Cannot get the list of
peersorg.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/replication/peersat
org.apache.zookeeper.KeeperException.create(KeeperException.java:130)at
org.apache.zookeeper.KeeperException.create(KeeperException.java:54)at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1549)at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:312)at
org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513)at
org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.getAllPeerConfigs(ReplicationPeersZKImpl.java:249)at
org.apache.hadoop.hbase.client.replication.ReplicationAdmin.listPeerConfigs(ReplicationAdmin.java:332)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
Apache-HBase commented on pull request #2044: URL: https://github.com/apache/hbase/pull/2044#issuecomment-656863864 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 56s | master passed | | +1 :green_heart: | compile | 3m 9s | master passed | | +1 :green_heart: | shadedjars | 6m 43s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | root in master failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 40s | the patch passed | | +1 :green_heart: | compile | 3m 9s | the patch passed | | +1 :green_heart: | javac | 3m 9s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 49s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 15s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 263m 51s | root in the patch passed. | | | | 298m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2044 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 77fbe74d915b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/testReport/ | | Max. process+thread count | 4232 (vs. ulimit of 12500) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2014: HBASE-24673 TransitionRegionStateProcedure of non-meta regions should…
ndimiduk commented on a change in pull request #2014:
URL: https://github.com/apache/hbase/pull/2014#discussion_r453024239
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
##
@@ -200,14 +200,21 @@ private void queueAssign(MasterProcedureEnv env,
RegionStateNode regionNode)
}
}
- private void openRegion(MasterProcedureEnv env, RegionStateNode regionNode)
throws IOException {
+ private void openRegion(MasterProcedureEnv env, RegionStateNode regionNode)
+throws IOException, ProcedureSuspendedException {
ServerName loc = regionNode.getRegionLocation();
if (loc == null) {
LOG.warn("No location specified for {}, jump back to state {} to get
one", getRegion(),
RegionStateTransitionState.REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE);
setNextState(RegionStateTransitionState.REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE);
return;
}
+final boolean isMeta = regionNode.getRegionInfo().isMetaRegion();
+final boolean isMetaAvailable =
!env.getAssignmentManager().isMetaRegionInTransition();
+if (!isMeta && !isMetaAvailable) {
+ // meta is not assigned yet, so yield
+ throw new ProcedureSuspendedException();
Review comment:
Hmm maybe. This part of the code is not very chatty, and this is kind of
an expected scenario. Maybe DEBUG level.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
##
@@ -200,14 +200,21 @@ private void queueAssign(MasterProcedureEnv env,
RegionStateNode regionNode)
}
}
- private void openRegion(MasterProcedureEnv env, RegionStateNode regionNode)
throws IOException {
+ private void openRegion(MasterProcedureEnv env, RegionStateNode regionNode)
+throws IOException, ProcedureSuspendedException {
ServerName loc = regionNode.getRegionLocation();
if (loc == null) {
LOG.warn("No location specified for {}, jump back to state {} to get
one", getRegion(),
RegionStateTransitionState.REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE);
setNextState(RegionStateTransitionState.REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE);
return;
}
+final boolean isMeta = regionNode.getRegionInfo().isMetaRegion();
+final boolean isMetaAvailable =
!env.getAssignmentManager().isMetaRegionInTransition();
+if (!isMeta && !isMetaAvailable) {
Review comment:
After more reading, I think this approach is wrong. There's already a
facility for waiting on meta to be available. My ITBLL experience indicates it
doesn't go far enough. I found:
```
@Override
protected boolean waitInitialized(MasterProcedureEnv env) {
if (TableName.isMetaTableName(getTableName())) {
return false;
}
// First we need meta to be loaded, and second, if meta is not online
then we will likely to
// fail when updating meta so we wait until it is assigned.
AssignmentManager am = env.getAssignmentManager();
return am.waitMetaLoaded(this) || am.waitMetaAssigned(this, getRegion());
}
```
https://github.com/apache/hbase/blob/809a623/hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java#L168-L176
Meta has already been assigned by the time this code executes, it's simply
not coming online. Let me poke around some more on the region server side, see
if I can make sense from that angle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
joshelser commented on pull request #1935: URL: https://github.com/apache/hbase/pull/1935#issuecomment-656832955 > My change removes these QUOTA_FAMILY_USAGE column family for the tables when a namespace quota is deleted and allows the user to insert into the tables in the namespace when the violated space quota is removed. > But I thought you should fix the problem not in QuotaUtil class, maybe in MasterQuotaManager? Keep QuotaUtil simple and fix this in upper layer. Thanks for the explanation in https://github.com/apache/hbase/pull/1935#pullrequestreview-439900394, Surbhi. This helps tremendously. I'm not sure that removing the `u:p` cell is the correct solution. These columns are just tracking the _size_ of the table. In themselves, they aren't controlling anything (if memory serves). Instead, we should know that when we delete the NamespaceQuota, MasterQuotaManager needs to become aware of this (somehow) and remove any SpaceQuotaViolationPolicies that are in effect. I suspect your fix does work, but for the wrong reason. If you remove the sizes for these tables, the Master will simply think that the table is no longer in validation. If you have a table-level quota also defined (like I commented in-code), I think this change will incorrectly cause that table-level to not be applied. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on a change in pull request #1935: HBASE-22146 SpaceQuotaViolationPolicy Disable is not working in Names…
joshelser commented on a change in pull request #1935:
URL: https://github.com/apache/hbase/pull/1935#discussion_r453019391
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
##
@@ -264,7 +265,32 @@ private static void deleteQuotas(final Connection
connection, final byte[] rowKe
final byte[] qualifier) throws IOException {
Delete delete = new Delete(rowKey);
if (qualifier != null) {
- delete.addColumns(QUOTA_FAMILY_INFO, qualifier);
+ //Check if delete qualifier is for persisted space quota snapshot usage
column family
+ if (Arrays.equals(qualifier,QUOTA_QUALIFIER_POLICY)) {
+delete.addColumns(QUOTA_FAMILY_USAGE, qualifier);
+ } else
+delete.addColumns(QUOTA_FAMILY_INFO, qualifier);
+}
+if (isNamespaceRowKey(rowKey)) {
+ //Check namespace is not deleted before you get info about quota and
list of tables in namespace
+ NamespaceDescriptor[] descs =
connection.getAdmin().listNamespaceDescriptors();
Review comment:
Beware that this requires ADMIN to list descriptors. This might be
increasing the permissions required to modify quotas.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
##
@@ -264,7 +265,32 @@ private static void deleteQuotas(final Connection
connection, final byte[] rowKe
final byte[] qualifier) throws IOException {
Delete delete = new Delete(rowKey);
if (qualifier != null) {
- delete.addColumns(QUOTA_FAMILY_INFO, qualifier);
+ //Check if delete qualifier is for persisted space quota snapshot usage
column family
+ if (Arrays.equals(qualifier,QUOTA_QUALIFIER_POLICY)) {
+delete.addColumns(QUOTA_FAMILY_USAGE, qualifier);
+ } else
Review comment:
Missing curly-brackets on else-branch
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaUtil.java
##
@@ -264,7 +265,32 @@ private static void deleteQuotas(final Connection
connection, final byte[] rowKe
final byte[] qualifier) throws IOException {
Delete delete = new Delete(rowKey);
if (qualifier != null) {
- delete.addColumns(QUOTA_FAMILY_INFO, qualifier);
+ //Check if delete qualifier is for persisted space quota snapshot usage
column family
+ if (Arrays.equals(qualifier,QUOTA_QUALIFIER_POLICY)) {
+delete.addColumns(QUOTA_FAMILY_USAGE, qualifier);
+ } else
+delete.addColumns(QUOTA_FAMILY_INFO, qualifier);
+}
+if (isNamespaceRowKey(rowKey)) {
+ //Check namespace is not deleted before you get info about quota and
list of tables in namespace
+ NamespaceDescriptor[] descs =
connection.getAdmin().listNamespaceDescriptors();
+ String ns = getNamespaceFromRowKey(rowKey);
+ int index = 0;
+ while (index < descs.length) {
+if (ns.equals(descs[index].getName())) {
+ Quotas namespaceQuota = getNamespaceQuota(connection,ns);
+ if (namespaceQuota != null && namespaceQuota.hasSpace()) {
+TableName[] tableArray =
connection.getAdmin().listTableNamesByNamespace(ns);
+for (TableName tableName : tableArray) {
+ deleteQuotas(connection, getTableRowKey(tableName),
QUOTA_QUALIFIER_POLICY);
Review comment:
If I override the namespace quota with a table-level quota, will this
remove my table-level quota too?
e.g. say there is a 10G quota on a namespace, but I further restrict a
specific table in that namespace to be 1G. I would expect that my table can
only use 1G at most.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-20552) HBase RegionServer was shutdown due to UnexpectedStateException
[
https://issues.apache.org/jira/browse/HBASE-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155669#comment-17155669
]
Josh Elser commented on HBASE-20552:
[~shenshengli], what version are you on, by chance? I haven't seen this
recently.
> HBase RegionServer was shutdown due to UnexpectedStateException
> ---
>
> Key: HBASE-20552
> URL: https://issues.apache.org/jira/browse/HBASE-20552
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Romil Choksi
>Assignee: Umesh Agashe
>Priority: Critical
> Attachments:
> 102143-master-ctr-e138-1518143905142-279227-01-03.hwx.site.log,
> 102143-master-ctr-e138-1518143905142-279227-01-05.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-02.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-07.hwx.site.log,
> 102143-regionserver-ctr-e138-1518143905142-279227-01-08.hwx.site.log
>
>
> This was observed during cluster testing (source code sync'ed with hbase-2.0,
> built May 2nd):
> {code}
> 2018-05-02 05:44:10,089 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=2]
> master.MasterRpcServices: Region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474 reported
> a fatal error:
> * ABORTING region server
> ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=ctr-e138-
> 1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on server=ctr-e138-
> 1518143905142-279227-01-02.hwx.site,16020,1525239334474 but state has
> otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1037)
> ... 7 more
> *
> Cause:
> org.apache.hadoop.hbase.YouAreDeadException:
> org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
>table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:459)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException:
> rit=OPEN,
> location=ctr-e138-1518143905142-279227-01-07.hwx.site,16020,1525239609353,
> table=test_hbase_ha_load_test_tool_hbase,
> region=94f6ca283dbb4445b2bcdc321b734d28reported OPEN on
> server=ctr-e138-1518143905142-279227-01-02.hwx.site,16020,1525239334474
> but state has otherwise.
> at
>
[jira] [Commented] (HBASE-24673) TransitionRegionStateProcedure of non-meta regions should yield when meta is unavailable
[
https://issues.apache.org/jira/browse/HBASE-24673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155666#comment-17155666
]
Nick Dimiduk commented on HBASE-24673:
--
Here's a stack trace collected from a stuck PEWorker thread, taken at the point
when the worker pool is expanding.
{noformat}
Thread 175 (PEWorker-1):
State: TIMED_WAITING
Blocked count: 3136
Waited count: 4122
Stack:
java.base@11.0.6/java.lang.Object.wait(Native Method)
app//org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:168)
app//org.apache.hadoop.hbase.client.HTable.put(HTable.java:539)
app//org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:224)
app//org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateUserRegionLocation(RegionStateStore.java:218)
app//org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:156)
app//org.apache.hadoop.hbase.master.assignment.AssignmentManager.transitStateAndUpdate(AssignmentManager.java:1743)
app//org.apache.hadoop.hbase.master.assignment.AssignmentManager.regionOpening(AssignmentManager.java:1758)
app//org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.openRegion(TransitRegionStateProcedure.java:211)
app//org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:357)
app//org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:102)
app//org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
app//org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:324)
app//org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:102)
app//org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1667)
app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1414)
app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:77)
app//org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1984)
{noformat}
> TransitionRegionStateProcedure of non-meta regions should yield when meta is
> unavailable
>
>
> Key: HBASE-24673
> URL: https://issues.apache.org/jira/browse/HBASE-24673
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> One observation from HBASE-24526 is that while meta is unavailable, other
> region movement procedures are getting stuck on meta RPCs. Let's make it so
> that non-meta transitions check the state of meta before attempting any RPCs.
> If meta is known unavailable, release the thread back to the scheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2006: HBASE-24632 Enable procedure-based log splitting as default in hbase3
Apache-HBase commented on pull request #2006: URL: https://github.com/apache/hbase/pull/2006#issuecomment-656817595 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 40s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 45s | branch-2 passed | | +1 :green_heart: | spotbugs | 3m 19s | branch-2 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 25s | the patch passed | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 6 new + 63 unchanged - 24 fixed = 69 total (was 87) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 4m 18s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | | | | 42m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2006/10/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2006 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 216c0fe084cb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ff2951e672 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2006/10/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-zookeeper hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2006/10/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
ndimiduk commented on a change in pull request #2017:
URL: https://github.com/apache/hbase/pull/2017#discussion_r452996830
##
File path:
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
##
@@ -1842,7 +1842,7 @@ private void
countDownChildren(RootProcedureState procStack,
store.update(parent);
scheduler.addFront(parent);
LOG.info("Finished subprocedure pid={}, resume processing parent {}",
- procedure.getProcId(), parent);
+ procedure.getProcId(), parent.toString().replace("pid=","ppid="));
Review comment:
Other attributes of the parent should be available via earlier log
lines; the operator can search for that `pid=` in the same log.
I still don't like doing a string replace on the output of another class's
`toString` method. A format string like `info("... parent ({})", parent);`
works as a compromise for me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2017: HBASE-24669 Logging of ppid should be consistent across all occurrences
ndimiduk commented on a change in pull request #2017:
URL: https://github.com/apache/hbase/pull/2017#discussion_r452996830
##
File path:
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
##
@@ -1842,7 +1842,7 @@ private void
countDownChildren(RootProcedureState procStack,
store.update(parent);
scheduler.addFront(parent);
LOG.info("Finished subprocedure pid={}, resume processing parent {}",
- procedure.getProcId(), parent);
+ procedure.getProcId(), parent.toString().replace("pid=","ppid="));
Review comment:
Other attributes of the parent should be available via earlier log
lines; the operator can search for that `pid=` in the same log.
I still don't like doing a string replace on the output of another class's
`toString` method. A format string like `info("... resume processing parent
({})", parent);` works as a compromise for me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] ndimiduk commented on a change in pull request #2034: Backport "HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970)" to branch
ndimiduk commented on a change in pull request #2034:
URL: https://github.com/apache/hbase/pull/2034#discussion_r452994871
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncProtobufLogWriter.java
##
@@ -231,4 +231,9 @@ protected long writeWALTrailerAndMagic(WALTrailer trailer,
byte[] magic) throws
protected OutputStream getOutputStreamForCellEncoder() {
return asyncOutputWrapper;
}
+
+ @Override
+ public long getSyncedLength() {
+return this.output.getSyncedLength();
Review comment:
Yep, understood. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24404) Support flush a single column family of region
[ https://issues.apache.org/jira/browse/HBASE-24404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155625#comment-17155625 ] Wellington Chevreuil commented on HBASE-24404: -- Thanks for the contribution [~filtertip], had merged it on master. Could you create a branch-2 compatible PR? > Support flush a single column family of region > -- > > Key: HBASE-24404 > URL: https://issues.apache.org/jira/browse/HBASE-24404 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 3.0.0-alpha-1 >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > It is useful when we set many column families for a table, could avoid > generate too many small files. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24404) Support flush a single column family of region
[ https://issues.apache.org/jira/browse/HBASE-24404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-24404: - Affects Version/s: 3.0.0-alpha-1 > Support flush a single column family of region > -- > > Key: HBASE-24404 > URL: https://issues.apache.org/jira/browse/HBASE-24404 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 3.0.0-alpha-1 >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > > It is useful when we set many column families for a table, could avoid > generate too many small files. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil merged pull request #2032: HBASE-24404 Support flush a single column family of region
wchevreuil merged pull request #2032: URL: https://github.com/apache/hbase/pull/2032 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on pull request #2032: HBASE-24404 Support flush a single column family of region
wchevreuil commented on pull request #2032: URL: https://github.com/apache/hbase/pull/2032#issuecomment-656782840 > The javadoc issues also exist in master, is this a bug of jdk11? See https://bugs.openjdk.java.net/browse/JDK-8212233. > > There is a workaround in project commons-pool, can we do as same way? > https://github.com/apache/commons-pool/blob/master/pom.xml > > ``` > > java11+ > > [11,) > > > > true > > > ``` > > Thanks. @wchevreuil No, let's leave as it is. Am going to merge it now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] petersomogyi commented on pull request #2046: HBASE-24711 Previous RC Release notes are not removed
petersomogyi commented on pull request #2046: URL: https://github.com/apache/hbase/pull/2046#issuecomment-656777345 > 臘 Exactly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24697) Source checksum is missing from web ui
[ https://issues.apache.org/jira/browse/HBASE-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24697. -- Resolution: Duplicate Closing this one since there's been some analysis on HBASE-24710. > Source checksum is missing from web ui > -- > > Key: HBASE-24697 > URL: https://issues.apache.org/jira/browse/HBASE-24697 > Project: HBase > Issue Type: Bug > Components: UI >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Priority: Minor > > The web ui has a "Software Attribute" section with a table entry called > "HBase Source Checksum", supposed to be a sha512 checksum according to the > description. Instead, the value is just "(stdin)=". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on pull request #2011: HBASE-24664 Some changing of split region by overall region size rath…
wchevreuil commented on pull request #2011: URL: https://github.com/apache/hbase/pull/2011#issuecomment-656771603 restest build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24715) Cleanup RELEASENOTES.md in the wake of HBASE-24711
[ https://issues.apache.org/jira/browse/HBASE-24715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-24715: - Affects Version/s: 2.3.0 > Cleanup RELEASENOTES.md in the wake of HBASE-24711 > -- > > Key: HBASE-24715 > URL: https://issues.apache.org/jira/browse/HBASE-24715 > Project: HBase > Issue Type: Sub-task > Components: community >Affects Versions: 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Seems it'll need some manual adjustment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24715) Cleanup RELEASENOTES.md in the wake of HBASE-24711
Nick Dimiduk created HBASE-24715: Summary: Cleanup RELEASENOTES.md in the wake of HBASE-24711 Key: HBASE-24715 URL: https://issues.apache.org/jira/browse/HBASE-24715 Project: HBase Issue Type: Sub-task Components: community Reporter: Nick Dimiduk Assignee: Nick Dimiduk Seems it'll need some manual adjustment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24586) Add table level locality in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-24586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155602#comment-17155602 ] Zheng Wang commented on HBASE-24586: No prob, will do it tomorrow, thanks.[~vjasani] > Add table level locality in table.jsp > - > > Key: HBASE-24586 > URL: https://issues.apache.org/jira/browse/HBASE-24586 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: HBASE-24586-afterpatch.png, HBASE-24586-beforepatch.png > > > For now, we only show locality for each region, it should be apply to table > as what Base Stats and Compactions does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on pull request #2046: HBASE-24711 Previous RC Release notes are not removed
ndimiduk commented on pull request #2046: URL: https://github.com/apache/hbase/pull/2046#issuecomment-656768765 臘 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24586) Add table level locality in table.jsp
[ https://issues.apache.org/jira/browse/HBASE-24586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155599#comment-17155599 ] Viraj Jasani commented on HBASE-24586: -- PR is merged in trunk. Due to changes in table.jsp b/ trunk and branch-2, can you raise PR for branch-2? Thanks [~filtertip] for your contribution. > Add table level locality in table.jsp > - > > Key: HBASE-24586 > URL: https://issues.apache.org/jira/browse/HBASE-24586 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Attachments: HBASE-24586-afterpatch.png, HBASE-24586-beforepatch.png > > > For now, we only show locality for each region, it should be apply to table > as what Base Stats and Compactions does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] wchevreuil commented on pull request #2009: HBASE-21596 Delete for a specific cell version can bring back version…
wchevreuil commented on pull request #2009: URL: https://github.com/apache/hbase/pull/2009#issuecomment-656767138 > > The only way to completely solve all the related problems is HBASE-15968, where we also consider MVCC so earlier modification will not effect later modifications. > > Agreed. But we need more test and usage case for this feature. @wchevreuil Do you meet same case for this problem? You can take a try for it. It will be better if we can foucs on one same solution. And for the document, I am +1 to update it and make it more clearly. I agree we should aim towards a unique solution. Is HBASE-15968 not considered production ready because of performance concerns? And since "versions" feature behaves inconsistently without HBASE-15968, shouldn't we recall it as not production ready as well? I can work on documenting the inconsistencies caused by versions and related delete markers, meanwhile. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani closed pull request #1926: HBASE-24586 Add table level locality in table.jsp
virajjasani closed pull request #1926: URL: https://github.com/apache/hbase/pull/1926 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24710) Incorrect checksum calculation in create-release Dockerfile
[ https://issues.apache.org/jira/browse/HBASE-24710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155596#comment-17155596 ] Peter Somogyi commented on HBASE-24710: --- I've checked in the Dockerfile and openssl returns the following when stdin is used: root@8224f4c69e6f:~# echo "foo" | openssl dgst -sha512 (stdin)= 0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6 On Mac this is the result: $ echo "foo" | openssl dgst -sha512 0cf9180a764aba863a67b6d72f0918bc131c6772642cb2dce5a34f0a702f9470ddc2bf125c12198b1995c233c34b4afd346c54a2334c350a948a51b6e8b4e6b6 > Incorrect checksum calculation in create-release Dockerfile > --- > > Key: HBASE-24710 > URL: https://issues.apache.org/jira/browse/HBASE-24710 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Peter Somogyi >Priority: Major > > The saveVersion.sh file does not parse the srcChecksum correctly during the > releasing process when dev-support/create-release/hbase-rm/Dockerfile is > used. This results in missing Source Checksum. > Master UI displays this: > |HBase Source Checksum|(stdin)=| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24711) Previous RC Release notes are not removed
[ https://issues.apache.org/jira/browse/HBASE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-24711: -- Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~vjasani] for the quick review. Note: It might require manual removal where the RELEASENOTES.md file already contains multiple entries for the same release. > Previous RC Release notes are not removed > - > > Key: HBASE-24711 > URL: https://issues.apache.org/jira/browse/HBASE-24711 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0-alpha-1 > > > The update_releasenotes function does not remove previous RC's Release notes > from RELEASENOTES.md file. > CHANGES.md does not have this issue. > [https://github.com/apache/hbase/blob/master/dev-support/create-release/release-util.sh#L493-L497] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi merged pull request #2046: HBASE-24711 Previous RC Release notes are not removed
petersomogyi merged pull request #2046: URL: https://github.com/apache/hbase/pull/2046 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2046: HBASE-24711 Previous RC Release notes are not removed
Apache-HBase commented on pull request #2046: URL: https://github.com/apache/hbase/pull/2046#issuecomment-656763201 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 54s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 2m 32s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2046/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2046 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 4af9d7935ab2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f0bc33122e | | Max. process+thread count | 49 (vs. ulimit of 12500) | | modules | C: U: | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2046/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2046: HBASE-24711 Previous RC Release notes are not removed
Apache-HBase commented on pull request #2046: URL: https://github.com/apache/hbase/pull/2046#issuecomment-656763432 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2005: HBASE-24662 Update DumpClusterStatusAction to notice changes in regio…
virajjasani commented on a change in pull request #2005:
URL: https://github.com/apache/hbase/pull/2005#discussion_r452940628
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/DumpClusterStatusAction.java
##
@@ -28,18 +35,69 @@
public class DumpClusterStatusAction extends Action {
private static final Logger LOG =
LoggerFactory.getLogger(DumpClusterStatusAction.class);
- @Override protected Logger getLogger() {
+ private Set initialRegionServers;
+
+ @Override
+ protected Logger getLogger() {
return LOG;
}
@Override
public void init(ActionContext context) throws IOException {
super.init(context);
+initialRegionServers = collectKnownRegionServers(initialStatus);
}
@Override
public void perform() throws Exception {
getLogger().debug("Performing action: Dump cluster status");
-getLogger().info("Cluster status\n" + cluster.getClusterMetrics());
+final ClusterMetrics currentMetrics = cluster.getClusterMetrics();
+getLogger().info("Cluster status\n{}", currentMetrics);
+reportMissingRegionServers(currentMetrics);
+reportNewRegionServers(currentMetrics);
+ }
+
+ /**
+ * Build a set of all the host:port pairs of region servers known to this
cluster.
+ */
+ private static Set collectKnownRegionServers(final ClusterMetrics
clusterMetrics) {
+final Set regionServers = clusterMetrics.getLiveServerMetrics()
+ .keySet()
+ .stream()
+ .map(ServerName::getAddress)
+ .collect(Collectors.toSet());
+clusterMetrics.getDeadServerNames()
+ .stream()
+ .map(ServerName::getAddress)
+ .forEach(regionServers::add);
+return Collections.unmodifiableSet(regionServers);
+ }
+
+ private void reportMissingRegionServers(final ClusterMetrics clusterMetrics)
{
+final Set regionServers =
collectKnownRegionServers(clusterMetrics);
+final Set missingRegionServers = new
HashSet<>(initialRegionServers);
+missingRegionServers.removeAll(regionServers);
+if (!missingRegionServers.isEmpty()) {
+ final StringBuilder stringBuilder = new StringBuilder()
+.append("region server(s) are missing from this cluster report");
+ missingRegionServers.forEach(address -> {
+stringBuilder.append("\n ").append(address);
+ });
Review comment:
nit: lambda expression `address -> stringBuilder.append("\n
").append(address)`
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/DumpClusterStatusAction.java
##
@@ -28,18 +35,69 @@
public class DumpClusterStatusAction extends Action {
private static final Logger LOG =
LoggerFactory.getLogger(DumpClusterStatusAction.class);
- @Override protected Logger getLogger() {
+ private Set initialRegionServers;
+
+ @Override
+ protected Logger getLogger() {
return LOG;
}
@Override
public void init(ActionContext context) throws IOException {
super.init(context);
+initialRegionServers = collectKnownRegionServers(initialStatus);
}
@Override
public void perform() throws Exception {
getLogger().debug("Performing action: Dump cluster status");
-getLogger().info("Cluster status\n" + cluster.getClusterMetrics());
+final ClusterMetrics currentMetrics = cluster.getClusterMetrics();
+getLogger().info("Cluster status\n{}", currentMetrics);
+reportMissingRegionServers(currentMetrics);
+reportNewRegionServers(currentMetrics);
+ }
+
+ /**
+ * Build a set of all the host:port pairs of region servers known to this
cluster.
+ */
+ private static Set collectKnownRegionServers(final ClusterMetrics
clusterMetrics) {
+final Set regionServers = clusterMetrics.getLiveServerMetrics()
+ .keySet()
+ .stream()
+ .map(ServerName::getAddress)
+ .collect(Collectors.toSet());
+clusterMetrics.getDeadServerNames()
+ .stream()
+ .map(ServerName::getAddress)
+ .forEach(regionServers::add);
+return Collections.unmodifiableSet(regionServers);
+ }
+
+ private void reportMissingRegionServers(final ClusterMetrics clusterMetrics)
{
+final Set regionServers =
collectKnownRegionServers(clusterMetrics);
+final Set missingRegionServers = new
HashSet<>(initialRegionServers);
+missingRegionServers.removeAll(regionServers);
+if (!missingRegionServers.isEmpty()) {
+ final StringBuilder stringBuilder = new StringBuilder()
+.append("region server(s) are missing from this cluster report");
+ missingRegionServers.forEach(address -> {
+stringBuilder.append("\n ").append(address);
+ });
+ getLogger().warn(stringBuilder.toString());
+}
+ }
+
+ private void reportNewRegionServers(final ClusterMetrics clusterMetrics) {
+final Set regionServers =
collectKnownRegionServers(clusterMetrics);
+final Set newRegionServers = new HashSet<>(regionServers);
+newRegionServers.removeAll(initialRegionServers);
+if
[jira] [Comment Edited] (HBASE-24672) HBase Shell Commands Survey
[ https://issues.apache.org/jira/browse/HBASE-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155586#comment-17155586 ] Elliot Miller edited comment on HBASE-24672 at 7/10/20, 4:16 PM: - I added one more column (to the spreadsheet) for unit testing. This column indicates which shell commands are directly tested (ie. not indirectly covered). My initial results indicate that 47 / 162 shell commands are currently missing unit tests. was (Author: bitoffdev): I added one more column for unit testing. This column indicates which shell commands are directly tested (ie. not indirectly covered). My initial results indicate that 47 / 162 shell commands are currently missing unit tests. > HBase Shell Commands Survey > --- > > Key: HBASE-24672 > URL: https://issues.apache.org/jira/browse/HBASE-24672 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Elliot Miller >Assignee: Elliot Miller >Priority: Minor > > I am going through all 163 commands in the hbase-shell module and checking a > few things: > * Functions as advertised > * Consistent naming, formatting, and help > * Return values > ** The majority of the commands still return nil. We can make the shell more > powerful by switching some of these commands to return Ruby objects. > h3. Acceptance Criteria > * The product of this ticket will be a *spreadsheet* with my comments on > each command and potentially a design doc or other Jira tickets with the > necessary changes found by the survey. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24672) HBase Shell Commands Survey
[ https://issues.apache.org/jira/browse/HBASE-24672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155586#comment-17155586 ] Elliot Miller commented on HBASE-24672: --- I added one more column for unit testing. This column indicates which shell commands are directly tested (ie. not indirectly covered). My initial results indicate that 47 / 162 shell commands are currently missing unit tests. > HBase Shell Commands Survey > --- > > Key: HBASE-24672 > URL: https://issues.apache.org/jira/browse/HBASE-24672 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Elliot Miller >Assignee: Elliot Miller >Priority: Minor > > I am going through all 163 commands in the hbase-shell module and checking a > few things: > * Functions as advertised > * Consistent naming, formatting, and help > * Return values > ** The majority of the commands still return nil. We can make the shell more > powerful by switching some of these commands to return Ruby objects. > h3. Acceptance Criteria > * The product of this ticket will be a *spreadsheet* with my comments on > each command and potentially a design doc or other Jira tickets with the > necessary changes found by the survey. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21596) Delete for a specific cell version can bring back versions above VERSIONS limit
[ https://issues.apache.org/jira/browse/HBASE-21596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HBASE-21596: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Following discussions on [PR 2009|https://github.com/apache/hbase/pull/2009], we agreed this would not be fixed. HBASE-15968 proposes a configurable solution that can be enabled at CF definition, so applications using versions should resort to HBASE-15968. > Delete for a specific cell version can bring back versions above VERSIONS > limit > --- > > Key: HBASE-21596 > URL: https://issues.apache.org/jira/browse/HBASE-21596 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-1 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0-alpha-1 > > Attachments: HBASE-21596-master.001.patch, > HBASE-21596-master.002.patch, HBASE-21596-master.003.patch, initial-patch.txt > > > Originally tested with HBase Shell delete command, but it's also reproducible > with Client API Delete operation. > The problem is that the memstore scan filter logic for versions only counts > the amount of cells it has read so far, then once the VERSIONS limit has been > reached, it just skips the remaining cells. If a delete marker is inserted on > a given cell version, that cell will not be accounted, then oldest versions > that should had disappeared will now pop up on the scan results. > Example, for a cell from a CF with max versions of 3, that has 4 versions T1, > T2, T3 and T4, scan correctly shows T4, T3 and T2. If a delete is triggered > for any for these 3 versions, say T3, scan now will show: T4, T2 and T1, but > T1 was supposed to be gone by the time T4 was added. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24711) Previous RC Release notes are not removed
[ https://issues.apache.org/jira/browse/HBASE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-24711: -- Status: Patch Available (was: Open) > Previous RC Release notes are not removed > - > > Key: HBASE-24711 > URL: https://issues.apache.org/jira/browse/HBASE-24711 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > > The update_releasenotes function does not remove previous RC's Release notes > from RELEASENOTES.md file. > CHANGES.md does not have this issue. > [https://github.com/apache/hbase/blob/master/dev-support/create-release/release-util.sh#L493-L497] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #2046: HBASE-24711 Previous RC Release notes are not removed
petersomogyi opened a new pull request #2046: URL: https://github.com/apache/hbase/pull/2046 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
Apache-HBase commented on pull request #2044: URL: https://github.com/apache/hbase/pull/2044#issuecomment-656756793 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | checkstyle | 2m 16s | master passed | | +0 :ok: | refguide | 5m 20s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | spotbugs | 11m 50s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 41s | the patch passed | | +1 :green_heart: | checkstyle | 2m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +0 :ok: | refguide | 5m 32s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | hadoopcheck | 12m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 12m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 69m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2044 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle refguide | | uname | Linux aaa909b985ca 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-general-check/output/branch-site/book.html | | refguide | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/artifact/yetus-general-check/output/patch-site/book.html | | Max. process+thread count | 122 (vs. ulimit of 12500) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2037: HBASE-24698 Turn OFF Canary WebUI as default
saintstack commented on pull request #2037: URL: https://github.com/apache/hbase/pull/2037#issuecomment-656756543 @Apache9 I was going to merge unless objection... I put back the define that is still used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil closed pull request #2036: HBASE-21596 Delete for a specific cell version can bring back version…
wchevreuil closed pull request #2036: URL: https://github.com/apache/hbase/pull/2036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-24711) Previous RC Release notes are not removed
[ https://issues.apache.org/jira/browse/HBASE-24711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi reassigned HBASE-24711: - Assignee: Peter Somogyi > Previous RC Release notes are not removed > - > > Key: HBASE-24711 > URL: https://issues.apache.org/jira/browse/HBASE-24711 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > > The update_releasenotes function does not remove previous RC's Release notes > from RELEASENOTES.md file. > CHANGES.md does not have this issue. > [https://github.com/apache/hbase/blob/master/dev-support/create-release/release-util.sh#L493-L497] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #2000: HBASE-24658 Update PolicyBasedChaosMonkey to handle uncaught exceptions
virajjasani commented on a change in pull request #2000:
URL: https://github.com/apache/hbase/pull/2000#discussion_r452936475
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/monkies/PolicyBasedChaosMonkey.java
##
@@ -60,19 +66,30 @@ public PolicyBasedChaosMonkey(IntegrationTestingUtility
util, Collection
}
public PolicyBasedChaosMonkey(Properties monkeyProps,
IntegrationTestingUtility util,
-Policy... policies) {
-this.monkeyProps = monkeyProps;
-this.util = util;
-this.policies = policies;
+Collection policies) {
+this(monkeyProps, util, policies.toArray(new Policy[0]));
}
public PolicyBasedChaosMonkey(Properties monkeyProps,
IntegrationTestingUtility util,
-Collection policies) {
+Policy... policies) {
this.monkeyProps = monkeyProps;
-this.util = util;
-this.policies = policies.toArray(new Policy[policies.size()]);
+this.util = Objects.requireNonNull(util);
+this.policies = Objects.requireNonNull(policies);
+if (policies.length == 0) {
+ throw new IllegalArgumentException("policies may not be empty");
+}
+this.monkeyThreadPool = buildMonkeyThreadPool(policies.length);
}
+ private static ExecutorService buildMonkeyThreadPool(final int size) {
+return Executors.newFixedThreadPool(size, new ThreadFactoryBuilder()
Review comment:
This one actually sets daemon to true, we don't want that so maybe we
can have separate method for setting daemon to false.
However, this work could be separate Jira and if we have majority, maybe we
can make all Executor pool in HBase use our internal ThreadFactory and not from
guava.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155573#comment-17155573
]
Michael Stack commented on HBASE-11288:
---
{quote}You mean that, with 'general root table', we will also only expose a
very simple API for locating meta?
{quote}
Yes. We could do this.
{quote}Client will not go to the root region directly but the cache on master
and backup master?
{quote}
Cache tier implementation is not part of this discussion unless you think it
needs to be. That the API locating ROOT content is simple and so easy to back
w/ a general cache is sufficient to our purposes here? Or do you think we need
to say more on this topic? (As to the cache implementation, I'd think we'd want
to avoid involving Masters in any caching solution if ROOT is located elsewhere
– involving Masters would undo a 'pro' of the 'general root table' direction as
you point out subsequently in your comment).
Your addition to the design doc seems good – I undid the bolded red (smile) –
but it seems we can't have this new addition and the point above it both stand
at the same time: revisit?
While extra-tier vs Master-in-line-for-meta-lookups are main pivot point
figuring where to locate ROOT, there are others like Master is writing an
in-process Region rather than remote RPC'ing and it'll have no 'noisy
neighbors'... I updated the design.
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] virajjasani commented on a change in pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
virajjasani commented on a change in pull request #2044:
URL: https://github.com/apache/hbase/pull/2044#discussion_r452910042
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -827,26 +828,34 @@ protected double scale(double min, double max, double
value) {
*/
static class MoveCostFunction extends CostFunction {
private static final String MOVE_COST_KEY =
"hbase.master.balancer.stochastic.moveCost";
+private static final String MOVE_COST_OFFPEAK_KEY =
"hbase.master.balancer.stochastic.moveCost.offpeak";
private static final String MAX_MOVES_PERCENT_KEY =
"hbase.master.balancer.stochastic.maxMovePercent";
-private static final float DEFAULT_MOVE_COST = 7;
+@VisibleForTesting
+protected static final float DEFAULT_MOVE_COST = 7;
+protected static final float DEFAULT_MOVE_COST_OFFPEAK = 3;
private static final int DEFAULT_MAX_MOVES = 600;
private static final float DEFAULT_MAX_MOVE_PERCENT = 0.25f;
private final float maxMovesPercent;
+private final Configuration conf;
MoveCostFunction(Configuration conf) {
super(conf);
-
- // Move cost multiplier should be the same cost or higher than the rest
of the costs to ensure
- // that large benefits are need to overcome the cost of a move.
- this.setMultiplier(conf.getFloat(MOVE_COST_KEY, DEFAULT_MOVE_COST));
+ this.conf = conf;
// What percent of the number of regions a single run of the balancer
can move.
maxMovesPercent = conf.getFloat(MAX_MOVES_PERCENT_KEY,
DEFAULT_MAX_MOVE_PERCENT);
}
@Override
protected double cost() {
+ // Move cost multiplier should be the same cost or higher than the rest
of the costs to ensure
+ // that large benefits are need to overcome the cost of a move.
+ if (OffPeakHours.getInstance(conf).isOffPeakHour()) {
+this.setMultiplier(conf.getFloat(MOVE_COST_OFFPEAK_KEY,
DEFAULT_MOVE_COST_OFFPEAK));
+ } else {
+this.setMultiplier(conf.getFloat(MOVE_COST_KEY, DEFAULT_MOVE_COST));
+ }
Review comment:
Good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HBASE-24712) [Flaky] TestMasterNoCluster.testStopDuringStart on master
[ https://issues.apache.org/jira/browse/HBASE-24712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24712: - Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Forward-ported to trunk. Will monitor this test in flaky dashboard for some days. > [Flaky] TestMasterNoCluster.testStopDuringStart on master > - > > Key: HBASE-24712 > URL: https://issues.apache.org/jira/browse/HBASE-24712 > Project: HBase > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0-alpha-1 > > > We need to forward port HBASE-24656 to master branch. > The test logs from flaky report: > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6537/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6530/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > > 2020-07-10 07:08:04,815 DEBUG [Time-limited test-EventThread] > zookeeper.ZKWatcher(555): @Before-0x1010270f65d0001, quorum=127.0.0.1:59817, > baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, > state=SyncConnected, path=/hbase/backup-masters 2020-07-10 07:08:04,824 DEBUG > [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(358): > master:44963-0x1010270f65d, quorum=127.0.0.1:59817, baseZNode=/hbase Set > watcher on existing > znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,44963,1594364881716 > 2020-07-10 07:08:04,827 INFO [Time-limited test] zookeeper.ZKUtil(1809): > multi exception: org.apache.zookeeper.KeeperException$NotEmptyException: > KeeperErrorCode = Directory not empty; running operations sequentially > (runSequentialOnMultiFailure=true); > > org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = > Directory not empty for /hbase/backup-masters at > org.apache.hadoop.hbase.master.TestMasterNoCluster.tearDown(TestMasterNoCluster.java:98) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] virajjasani merged pull request #2045: HBASE-24712 : Remove flakes from TestMasterNoCluster#testStopDuringStart
virajjasani merged pull request #2045: URL: https://github.com/apache/hbase/pull/2045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24712) [Flaky] TestMasterNoCluster.testStopDuringStart on master
[ https://issues.apache.org/jira/browse/HBASE-24712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155539#comment-17155539 ] Viraj Jasani commented on HBASE-24712: -- [~stack] Based on the recent flaky reports, the test failures resolved in parent Jira (branch-2 and branch-2.3) should also be applied to master branch. Raised PR and build is good, let me merge this and will monitor this test for some days. I am just forward-porting your patch with slight modification. Thanks > [Flaky] TestMasterNoCluster.testStopDuringStart on master > - > > Key: HBASE-24712 > URL: https://issues.apache.org/jira/browse/HBASE-24712 > Project: HBase > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > We need to forward port HBASE-24656 to master branch. > The test logs from flaky report: > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6537/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6530/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > > 2020-07-10 07:08:04,815 DEBUG [Time-limited test-EventThread] > zookeeper.ZKWatcher(555): @Before-0x1010270f65d0001, quorum=127.0.0.1:59817, > baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, > state=SyncConnected, path=/hbase/backup-masters 2020-07-10 07:08:04,824 DEBUG > [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(358): > master:44963-0x1010270f65d, quorum=127.0.0.1:59817, baseZNode=/hbase Set > watcher on existing > znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,44963,1594364881716 > 2020-07-10 07:08:04,827 INFO [Time-limited test] zookeeper.ZKUtil(1809): > multi exception: org.apache.zookeeper.KeeperException$NotEmptyException: > KeeperErrorCode = Directory not empty; running operations sequentially > (runSequentialOnMultiFailure=true); > > org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = > Directory not empty for /hbase/backup-masters at > org.apache.hadoop.hbase.master.TestMasterNoCluster.tearDown(TestMasterNoCluster.java:98) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
Apache-HBase commented on pull request #2021: URL: https://github.com/apache/hbase/pull/2021#issuecomment-656722232 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 12s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 7s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 8s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 205m 40s | hbase-server in the patch passed. | | | | 232m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2021 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 2136f7c1411b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/testReport/ | | Max. process+thread count | 3695 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
bsglz commented on a change in pull request #2044:
URL: https://github.com/apache/hbase/pull/2044#discussion_r452894660
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -827,26 +828,34 @@ protected double scale(double min, double max, double
value) {
*/
static class MoveCostFunction extends CostFunction {
private static final String MOVE_COST_KEY =
"hbase.master.balancer.stochastic.moveCost";
+private static final String MOVE_COST_OFFPEAK_KEY =
"hbase.master.balancer.stochastic.moveCost.offpeak";
Review comment:
Fixed, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
bsglz commented on a change in pull request #2044:
URL: https://github.com/apache/hbase/pull/2044#discussion_r452891195
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -827,26 +828,34 @@ protected double scale(double min, double max, double
value) {
*/
static class MoveCostFunction extends CostFunction {
private static final String MOVE_COST_KEY =
"hbase.master.balancer.stochastic.moveCost";
+private static final String MOVE_COST_OFFPEAK_KEY =
"hbase.master.balancer.stochastic.moveCost.offpeak";
private static final String MAX_MOVES_PERCENT_KEY =
"hbase.master.balancer.stochastic.maxMovePercent";
-private static final float DEFAULT_MOVE_COST = 7;
+@VisibleForTesting
+protected static final float DEFAULT_MOVE_COST = 7;
+protected static final float DEFAULT_MOVE_COST_OFFPEAK = 3;
private static final int DEFAULT_MAX_MOVES = 600;
private static final float DEFAULT_MAX_MOVE_PERCENT = 0.25f;
private final float maxMovesPercent;
+private final Configuration conf;
MoveCostFunction(Configuration conf) {
super(conf);
-
- // Move cost multiplier should be the same cost or higher than the rest
of the costs to ensure
- // that large benefits are need to overcome the cost of a move.
- this.setMultiplier(conf.getFloat(MOVE_COST_KEY, DEFAULT_MOVE_COST));
+ this.conf = conf;
// What percent of the number of regions a single run of the balancer
can move.
maxMovesPercent = conf.getFloat(MAX_MOVES_PERCENT_KEY,
DEFAULT_MAX_MOVE_PERCENT);
}
@Override
protected double cost() {
+ // Move cost multiplier should be the same cost or higher than the rest
of the costs to ensure
+ // that large benefits are need to overcome the cost of a move.
+ if (OffPeakHours.getInstance(conf).isOffPeakHour()) {
+this.setMultiplier(conf.getFloat(MOVE_COST_OFFPEAK_KEY,
DEFAULT_MOVE_COST_OFFPEAK));
+ } else {
+this.setMultiplier(conf.getFloat(MOVE_COST_KEY, DEFAULT_MOVE_COST));
+ }
Review comment:
The blancer only be initialized once on start, but we want the
multiplier changed at offpeak hours.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2045: HBASE-24712 : Remove flakes from TestMasterNoCluster#testStopDuringStart
Apache-HBase commented on pull request #2045: URL: https://github.com/apache/hbase/pull/2045#issuecomment-656710534 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 9s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | shadedjars | 6m 3s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 38s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 6m 4s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 202m 37s | hbase-server in the patch passed. | | | | 228m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2045 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1a9900895d56 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/testReport/ | | Max. process+thread count | 3737 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
bsglz commented on a change in pull request #2044:
URL: https://github.com/apache/hbase/pull/2044#discussion_r452880162
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
##
@@ -827,26 +828,34 @@ protected double scale(double min, double max, double
value) {
*/
static class MoveCostFunction extends CostFunction {
private static final String MOVE_COST_KEY =
"hbase.master.balancer.stochastic.moveCost";
+private static final String MOVE_COST_OFFPEAK_KEY =
"hbase.master.balancer.stochastic.moveCost.offpeak";
private static final String MAX_MOVES_PERCENT_KEY =
"hbase.master.balancer.stochastic.maxMovePercent";
-private static final float DEFAULT_MOVE_COST = 7;
+@VisibleForTesting
+protected static final float DEFAULT_MOVE_COST = 7;
+protected static final float DEFAULT_MOVE_COST_OFFPEAK = 3;
Review comment:
Good advice, will fix.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24653) Show snapshot owner on Master WebUI
[
https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155510#comment-17155510
]
Hudson commented on HBASE-24653:
Results for branch master
[build #1782 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Show snapshot owner on Master WebUI
> ---
>
> Key: HBASE-24653
> URL: https://issues.apache.org/jira/browse/HBASE-24653
> Project: HBase
> Issue Type: Improvement
>Reporter: Yi Mei
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: snapshot1.png, snapshot2.png
>
>
> Now Master UI shows lots of snapshot informations, and owner is also useful
> to find out who create this snapshot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24489) Rewrite TestClusterRestartFailover.test since namespace table is gone on on master
[
https://issues.apache.org/jira/browse/HBASE-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155509#comment-17155509
]
Hudson commented on HBASE-24489:
Results for branch master
[build #1782 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Rewrite TestClusterRestartFailover.test since namespace table is gone on on
> master
> --
>
> Key: HBASE-24489
> URL: https://issues.apache.org/jira/browse/HBASE-24489
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Duo Zhang
>Assignee: Sun Xin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> We still have this
> {code}
> // Find server that does not have hbase:namespace on it. This tests holds
> up SCPs. If it
> // holds up the server w/ hbase:namespace, the Master initialization will
> be held up
> // because this table is not online and test fails.
> for (JVMClusterUtil.RegionServerThread rst:
> UTIL.getHBaseCluster().getLiveRegionServerThreads()) {
> HRegionServer rs = rst.getRegionServer();
> if (rs.getRegions(TableName.NAMESPACE_TABLE_NAME).isEmpty()) {
> SERVER_FOR_TEST = rs.getServerName();
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-21596) Delete for a specific cell version can bring back versions above VERSIONS limit
[
https://issues.apache.org/jira/browse/HBASE-21596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155508#comment-17155508
]
Hudson commented on HBASE-21596:
Results for branch master
[build #1782 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Delete for a specific cell version can bring back versions above VERSIONS
> limit
> ---
>
> Key: HBASE-21596
> URL: https://issues.apache.org/jira/browse/HBASE-21596
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 3.0.0-alpha-1
>
> Attachments: HBASE-21596-master.001.patch,
> HBASE-21596-master.002.patch, HBASE-21596-master.003.patch, initial-patch.txt
>
>
> Originally tested with HBase Shell delete command, but it's also reproducible
> with Client API Delete operation.
> The problem is that the memstore scan filter logic for versions only counts
> the amount of cells it has read so far, then once the VERSIONS limit has been
> reached, it just skips the remaining cells. If a delete marker is inserted on
> a given cell version, that cell will not be accounted, then oldest versions
> that should had disappeared will now pop up on the scan results.
> Example, for a cell from a CF with max versions of 3, that has 4 versions T1,
> T2, T3 and T4, scan correctly shows T4, T3 and T2. If a delete is triggered
> for any for these 3 versions, say T3, scan now will show: T4, T2 and T1, but
> T1 was supposed to be gone by the time T4 was added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24663) Add procedure process time statistics UI
[
https://issues.apache.org/jira/browse/HBASE-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155511#comment-17155511
]
Hudson commented on HBASE-24663:
Results for branch master
[build #1782 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1663//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1782/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add procedure process time statistics UI
>
>
> Key: HBASE-24663
> URL: https://issues.apache.org/jira/browse/HBASE-24663
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Junhong Xu
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: screenshot-1.png
>
>
> Added in "Procedures & Locks" jsp.
> For the first version UI, we care about the process time of
> ServerCrashProcedure, TRSP, OpenRegionProcedure and CloseRegionProcedure.
> Plan to show the avg/P50/P90/min/max process time of these procedures.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2010: HBASE-24391 Implement meta split
Apache-HBase commented on pull request #2010: URL: https://github.com/apache/hbase/pull/2010#issuecomment-656686911 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 37s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 10s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | compile | 1m 46s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | shadedjars | 6m 14s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | HBASE-11288.splittable-meta passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 2m 8s | the patch passed | | +1 :green_heart: | javac | 2m 8s | the patch passed | | +1 :green_heart: | shadedjars | 7m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 27s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 32s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 24s | hbase-balancer in the patch passed. | | -1 :x: | unit | 234m 15s | hbase-server in the patch failed. | | | | 269m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2010 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux dd311fa1a871 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / a232d11f71 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/testReport/ | | Max. process+thread count | 2953 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-balancer hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
Apache-HBase commented on pull request #2021: URL: https://github.com/apache/hbase/pull/2021#issuecomment-656686420 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 32s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 24s | master passed | | +1 :green_heart: | compile | 1m 6s | master passed | | +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 44s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 6s | the patch passed | | +1 :green_heart: | javac | 1m 6s | the patch passed | | +1 :green_heart: | shadedjars | 5m 50s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 131m 34s | hbase-server in the patch passed. | | | | 158m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2021 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 223861dcd838 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/testReport/ | | Max. process+thread count | 4218 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2045: HBASE-24712 : Remove flakes from TestMasterNoCluster#testStopDuringStart
Apache-HBase commented on pull request #2045: URL: https://github.com/apache/hbase/pull/2045#issuecomment-656677216 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | master passed | | +1 :green_heart: | compile | 1m 5s | master passed | | +1 :green_heart: | shadedjars | 5m 48s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 43s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 3s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 5m 58s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 132m 18s | hbase-server in the patch passed. | | | | 158m 56s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2045 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8353e5f3138a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/testReport/ | | Max. process+thread count | 4158 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2019: HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from Re…
Apache-HBase commented on pull request #2019: URL: https://github.com/apache/hbase/pull/2019#issuecomment-656675539 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 34s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | HBASE-24666 passed | | +1 :green_heart: | compile | 1m 10s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 6m 51s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 46s | hbase-server in HBASE-24666 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | the patch passed | | +1 :green_heart: | compile | 1m 12s | the patch passed | | +1 :green_heart: | javac | 1m 12s | the patch passed | | +1 :green_heart: | shadedjars | 6m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 45s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 224m 27s | hbase-server in the patch passed. | | | | 255m 7s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2019 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 86581908c096 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / c2949719c5 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/testReport/ | | Max. process+thread count | 2976 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2010: HBASE-24391 Implement meta split
Apache-HBase commented on pull request #2010: URL: https://github.com/apache/hbase/pull/2010#issuecomment-656669104 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-11288.splittable-meta Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 40s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | compile | 1m 56s | HBASE-11288.splittable-meta passed | | +1 :green_heart: | shadedjars | 6m 20s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-balancer in HBASE-11288.splittable-meta failed. | | -0 :warning: | javadoc | 0m 26s | hbase-client in HBASE-11288.splittable-meta failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in HBASE-11288.splittable-meta failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 33s | the patch passed | | +1 :green_heart: | compile | 1m 57s | the patch passed | | +1 :green_heart: | javac | 1m 57s | the patch passed | | +1 :green_heart: | shadedjars | 6m 23s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 16s | hbase-balancer in the patch failed. | | -0 :warning: | javadoc | 0m 41s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 23s | hbase-balancer in the patch passed. | | -1 :x: | unit | 196m 44s | hbase-server in the patch failed. | | | | 230m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2010 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux cde48991af87 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-11288.splittable-meta / a232d11f71 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-balancer.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-balancer.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/testReport/ | | Max. process+thread count | 3275 (vs. ulimit of 12500) | | modules | C: hbase-client hbase-balancer hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2010/4/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on a change in pull request #2000: HBASE-24658 Update PolicyBasedChaosMonkey to handle uncaught exceptions
virajjasani commented on a change in pull request #2000:
URL: https://github.com/apache/hbase/pull/2000#discussion_r452815115
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/monkies/PolicyBasedChaosMonkey.java
##
@@ -38,14 +42,16 @@
private static final Logger LOG =
LoggerFactory.getLogger(PolicyBasedChaosMonkey.class);
Review comment:
nit: redundant
##
File path:
hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/monkies/PolicyBasedChaosMonkey.java
##
@@ -60,19 +66,30 @@ public PolicyBasedChaosMonkey(IntegrationTestingUtility
util, Collection
}
public PolicyBasedChaosMonkey(Properties monkeyProps,
IntegrationTestingUtility util,
-Policy... policies) {
-this.monkeyProps = monkeyProps;
-this.util = util;
-this.policies = policies;
+Collection policies) {
+this(monkeyProps, util, policies.toArray(new Policy[0]));
}
public PolicyBasedChaosMonkey(Properties monkeyProps,
IntegrationTestingUtility util,
-Collection policies) {
+Policy... policies) {
this.monkeyProps = monkeyProps;
-this.util = util;
-this.policies = policies.toArray(new Policy[policies.size()]);
+this.util = Objects.requireNonNull(util);
+this.policies = Objects.requireNonNull(policies);
+if (policies.length == 0) {
+ throw new IllegalArgumentException("policies may not be empty");
+}
+this.monkeyThreadPool = buildMonkeyThreadPool(policies.length);
}
+ private static ExecutorService buildMonkeyThreadPool(final int size) {
+return Executors.newFixedThreadPool(size, new ThreadFactoryBuilder()
Review comment:
Would you prefer using our internal
`Threads.newDaemonThreadFactory(String prefix, UncaughtExceptionHandler
handler)` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2044: HBASE-24709 Support MoveCostFunction use a lower multiplier in offpea…
Apache-HBase commented on pull request #2044: URL: https://github.com/apache/hbase/pull/2044#issuecomment-656658939 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 58s | master passed | | +1 :green_heart: | compile | 3m 11s | master passed | | +1 :green_heart: | shadedjars | 6m 41s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | root in master failed. | | -0 :warning: | javadoc | 0m 46s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 3m 16s | the patch passed | | +1 :green_heart: | javac | 3m 16s | the patch passed | | +1 :green_heart: | shadedjars | 6m 40s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 51s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 13s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 264m 10s | root in the patch passed. | | | | 299m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2044 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e640fab0ccef 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/testReport/ | | Max. process+thread count | 4406 (vs. ulimit of 12500) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2044/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work started] (HBASE-24712) [Flaky] TestMasterNoCluster.testStopDuringStart on master
[ https://issues.apache.org/jira/browse/HBASE-24712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24712 started by Viraj Jasani. > [Flaky] TestMasterNoCluster.testStopDuringStart on master > - > > Key: HBASE-24712 > URL: https://issues.apache.org/jira/browse/HBASE-24712 > Project: HBase > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > We need to forward port HBASE-24656 to master branch. > The test logs from flaky report: > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6537/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6530/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > > 2020-07-10 07:08:04,815 DEBUG [Time-limited test-EventThread] > zookeeper.ZKWatcher(555): @Before-0x1010270f65d0001, quorum=127.0.0.1:59817, > baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, > state=SyncConnected, path=/hbase/backup-masters 2020-07-10 07:08:04,824 DEBUG > [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(358): > master:44963-0x1010270f65d, quorum=127.0.0.1:59817, baseZNode=/hbase Set > watcher on existing > znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,44963,1594364881716 > 2020-07-10 07:08:04,827 INFO [Time-limited test] zookeeper.ZKUtil(1809): > multi exception: org.apache.zookeeper.KeeperException$NotEmptyException: > KeeperErrorCode = Directory not empty; running operations sequentially > (runSequentialOnMultiFailure=true); > > org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = > Directory not empty for /hbase/backup-masters at > org.apache.hadoop.hbase.master.TestMasterNoCluster.tearDown(TestMasterNoCluster.java:98) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24712) [Flaky] TestMasterNoCluster.testStopDuringStart on master
[ https://issues.apache.org/jira/browse/HBASE-24712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HBASE-24712: - Status: Patch Available (was: In Progress) > [Flaky] TestMasterNoCluster.testStopDuringStart on master > - > > Key: HBASE-24712 > URL: https://issues.apache.org/jira/browse/HBASE-24712 > Project: HBase > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > We need to forward port HBASE-24656 to master branch. > The test logs from flaky report: > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6537/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/6530/testReport/org.apache.hadoop.hbase.master/TestMasterNoCluster/testStopDuringStart/] > > 2020-07-10 07:08:04,815 DEBUG [Time-limited test-EventThread] > zookeeper.ZKWatcher(555): @Before-0x1010270f65d0001, quorum=127.0.0.1:59817, > baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, > state=SyncConnected, path=/hbase/backup-masters 2020-07-10 07:08:04,824 DEBUG > [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(358): > master:44963-0x1010270f65d, quorum=127.0.0.1:59817, baseZNode=/hbase Set > watcher on existing > znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,44963,1594364881716 > 2020-07-10 07:08:04,827 INFO [Time-limited test] zookeeper.ZKUtil(1809): > multi exception: org.apache.zookeeper.KeeperException$NotEmptyException: > KeeperErrorCode = Directory not empty; running operations sequentially > (runSequentialOnMultiFailure=true); > > org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = > Directory not empty for /hbase/backup-masters at > org.apache.hadoop.hbase.master.TestMasterNoCluster.tearDown(TestMasterNoCluster.java:98) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24714) Error message is displayed in the UI of table's compaction state if any region of that table is not open.
Sanjeet Nishad created HBASE-24714: -- Summary: Error message is displayed in the UI of table's compaction state if any region of that table is not open. Key: HBASE-24714 URL: https://issues.apache.org/jira/browse/HBASE-24714 Project: HBase Issue Type: Bug Components: UI Affects Versions: 2.2.3, master Reporter: Sanjeet Nishad Assignee: Sanjeet Nishad Attachments: ErrorMessageinUI.png When a region of a table is in states closed, merging, splitting, etc, the table's compaction state displays error stack trace. !ErrorMessageinUI.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2019: HBASE-24681 Remove the cache walsById/walsByIdRecoveredQueues from Re…
Apache-HBase commented on pull request #2019: URL: https://github.com/apache/hbase/pull/2019#issuecomment-656637038 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 35s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ HBASE-24666 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 34s | HBASE-24666 passed | | +1 :green_heart: | compile | 0m 56s | HBASE-24666 passed | | +1 :green_heart: | shadedjars | 5m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | HBASE-24666 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 29s | the patch passed | | +1 :green_heart: | compile | 0m 56s | the patch passed | | +1 :green_heart: | javac | 0m 56s | the patch passed | | +1 :green_heart: | shadedjars | 5m 34s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 142m 14s | hbase-server in the patch passed. | | | | 166m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2019 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0e1a963ab077 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | HBASE-24666 / c2949719c5 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/testReport/ | | Max. process+thread count | 4384 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2019/3/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
Apache-HBase commented on pull request #2021: URL: https://github.com/apache/hbase/pull/2021#issuecomment-656632846 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | master passed | | +1 :green_heart: | checkstyle | 1m 7s | master passed | | +1 :green_heart: | spotbugs | 2m 5s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 23s | the patch passed | | -0 :warning: | checkstyle | 1m 2s | hbase-server: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 59s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 35m 40s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2021 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux d6e713866427 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 94 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2021/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2045: HBASE-24712 : Remove flakes from TestMasterNoCluster#testStopDuringStart
Apache-HBase commented on pull request #2045: URL: https://github.com/apache/hbase/pull/2045#issuecomment-656628538 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 53s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 52s | master passed | | +1 :green_heart: | checkstyle | 1m 24s | master passed | | +1 :green_heart: | spotbugs | 2m 42s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 23s | the patch passed | | +1 :green_heart: | checkstyle | 1m 21s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 14s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 36s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 42m 54s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2045 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux cc533128cedc 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 67dfbe0406 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2045/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi closed pull request #2008: HBASE-24665 all wal of RegionGroupingProvider together roll
WenFeiYi closed pull request #2008: URL: https://github.com/apache/hbase/pull/2008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] WenFeiYi removed a comment on pull request #2008: HBASE-24665 all wal of RegionGroupingProvider together roll
WenFeiYi removed a comment on pull request #2008: URL: https://github.com/apache/hbase/pull/2008#issuecomment-655444997 > Can you add more comments to explain "why the fix worked"? And add a UT for this? I update, please review, if ok, I update other branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
