[jira] [Commented] (HBASE-23286) Improve MTTR: Split WAL to HFile
[ https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012543#comment-17012543 ] Guanghao Zhang commented on HBASE-23286: {quote}bq. I think [~zghao] has already ported the patch to our internal branch-2 based branch and tested the feature? {quote} Yes. I tested this on our internal branch. > Improve MTTR: Split WAL to HFile > > > Key: HBASE-23286 > URL: https://issues.apache.org/jira/browse/HBASE-23286 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > After HBASE-20724, the compaction event marker is not used anymore when > failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3 > steps: > # Read WAL and write HFile to region’s column family’s recovered.hfiles > directory. > # Open region. > # Bulkload the recovered.hfiles for every column family. > The design doc was attathed by a google doc. Any suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012552#comment-17012552 ] Guanghao Zhang commented on HBASE-23659: Pushed to branch-2.2+. Thanks all for reviewing. > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23659: --- Fix Version/s: 2.2.3 2.3.0 3.0.0 > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[ https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012557#comment-17012557 ] Szabolcs Bukros commented on HBASE-23601: - [~stack] Thanks for the merge! This doesn't need a master patch. The code there was heavily rewritten and as far as I can tell it's not affected by this issue. I'll check why it fails on branch-2. > OutputSink.WriterThread exception gets stuck and repeated indefinietly > -- > > Key: HBASE-23601 > URL: https://issues.apache.org/jira/browse/HBASE-23601 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.2.2 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4 > > > When a WriterThread runs into an exception (ie: NotServingRegionException), > the exception is stored in the controller. It is never removed and can not be > overwritten either. > > {code:java} > public void run() { > try { > doRun(); > } catch (Throwable t) { > LOG.error("Exiting thread", t); > controller.writerThreadError(t); > } > }{code} > Thanks to this every time PipelineController.checkForErrors() is called the > same old exception is rethrown. > > For example in RegionReplicaReplicationEndpoint.replicate there is a while > loop that does the actual replicating. Every time it loops, it calls > checkForErrors(), catches the rethrown exception, logs it but does nothing > about it. This results in ~2GB log files in ~5min in my experience. > > My proposal would be to clean up the stored exception when it reaches > RegionReplicaReplicationEndpoint.replicate and make sure we restart the > WriterThread that died throwing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23673) org.apache.hadoop.hbase.ipc.RpcServer.Listener#closeCurrentConnection did not close the connection properly
[ https://issues.apache.org/jira/browse/HBASE-23673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012560#comment-17012560 ] Baiqiang Zhao commented on HBASE-23673: --- Ping [~stack] > org.apache.hadoop.hbase.ipc.RpcServer.Listener#closeCurrentConnection did not > close the connection properly > --- > > Key: HBASE-23673 > URL: https://issues.apache.org/jira/browse/HBASE-23673 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Affects Versions: 1.6.0, 1.4.12 >Reporter: Baiqiang Zhao >Priority: Major > > The connection is attached to readKey, but the parameter passed into the > closeCurrentConnection is acceptedKey. branch-2 is fixed in HBASE-15948, > though 2.0+ already uses netty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23655. Resolution: Fixed > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23659. Resolution: Fixed > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[ https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23658. Resolution: Fixed > Fix flaky TestSnapshotFromMaster > > > Key: HBASE-23658 > URL: https://issues.apache.org/jira/browse/HBASE-23658 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > Attachments: HBASE-23658.branch-2.2.001.patch > > > testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may > fail. > {code:java} > assertTrue(master.getSnapshotManager().isTakingAnySnapshot()); > future.get(); // in branch-2.2, here is Thread.sleep > assertFalse(master.getSnapshotManager().isTakingAnySnapshot()); > {code} > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/] > > [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23663) Allow dot and hyphen in Profiler's URL
[ https://issues.apache.org/jira/browse/HBASE-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23663: --- Fix Version/s: (was: 2.2.4) 2.2.3 > Allow dot and hyphen in Profiler's URL > -- > > Key: HBASE-23663 > URL: https://issues.apache.org/jira/browse/HBASE-23663 > Project: HBase > Issue Type: Bug > Components: profiler >Affects Versions: 3.0.0, 2.2.0, 2.3.0, 2.1.5 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > ProfileOutputServlet sanitizes the input URL but fails when the query string > contains dot or hyphen. These are valid characters for the hostname. > Example URL part: > /prof-output/async-prof-pid-122466-cpu-1.svg?host=myhost-1.example.com&port=16010 > > In this case the user gets the following error message: > {noformat} > HTTP ERROR: 500 > Problem accessing /prof-output/async-prof-pid-122466-cpu-1.svg. Reason: > java.lang.RuntimeException: Non-alphanumeric data found in input, > aborting. > Powered by Jetty:// 9.3.27.v20190418 {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23651) Region balance throttling can be disabled
[ https://issues.apache.org/jira/browse/HBASE-23651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23651: --- Fix Version/s: (was: 2.2.4) 2.2.3 > Region balance throttling can be disabled > - > > Key: HBASE-23651 > URL: https://issues.apache.org/jira/browse/HBASE-23651 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > HBASE-17178 Add region balance throttling, but it can not be disabled, > sometimes we need no throttle and balance the cluster as fast as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23666) Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-23666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23666: --- Fix Version/s: (was: 2.2.3) > Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to > branch-2 > - > > Key: HBASE-23666 > URL: https://issues.apache.org/jira/browse/HBASE-23666 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.2 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23666) Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-23666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23666: --- Fix Version/s: 2.2.3 > Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to > branch-2 > - > > Key: HBASE-23666 > URL: https://issues.apache.org/jira/browse/HBASE-23666 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.2 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 2.3.0, 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[ https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012637#comment-17012637 ] Guanghao Zhang commented on HBASE-23601: I plan to release 2.2.3 now. Let me revert it from branch-2.2 as this not resolved. Will merge this after release. Thanks. > OutputSink.WriterThread exception gets stuck and repeated indefinietly > -- > > Key: HBASE-23601 > URL: https://issues.apache.org/jira/browse/HBASE-23601 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.2.2 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4 > > > When a WriterThread runs into an exception (ie: NotServingRegionException), > the exception is stored in the controller. It is never removed and can not be > overwritten either. > > {code:java} > public void run() { > try { > doRun(); > } catch (Throwable t) { > LOG.error("Exiting thread", t); > controller.writerThreadError(t); > } > }{code} > Thanks to this every time PipelineController.checkForErrors() is called the > same old exception is rethrown. > > For example in RegionReplicaReplicationEndpoint.replicate there is a while > loop that does the actual replicating. Every time it loops, it calls > checkForErrors(), catches the rethrown exception, logs it but does nothing > about it. This results in ~2GB log files in ~5min in my experience. > > My proposal would be to clean up the stored exception when it reaches > RegionReplicaReplicationEndpoint.replicate and make sure we restart the > WriterThread that died throwing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio merged pull request #992: HBASE-23637 Generate CHANGES.md and RELEASENOTES.md for 2.2.3 (addendum)
infraio merged pull request #992: HBASE-23637 Generate CHANGES.md and RELEASENOTES.md for 2.2.3 (addendum) URL: https://github.com/apache/hbase/pull/992 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #992: HBASE-23637 Generate CHANGES.md and RELEASENOTES.md for 2.2.3 (addendum)
Apache-HBase commented on issue #992: HBASE-23637 Generate CHANGES.md and RELEASENOTES.md for 2.2.3 (addendum) URL: https://github.com/apache/hbase/pull/992#issuecomment-572974195 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 7s | https://github.com/apache/hbase/pull/992 does not apply to branch-2.2. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/992 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-992/2/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1015: HBASE-23668 Master log start filling with "Flush journal status" messages
Apache-HBase commented on issue #1015: HBASE-23668 Master log start filling with "Flush journal status" messages URL: https://github.com/apache/hbase/pull/1015#issuecomment-572995230 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 52s | branch-2 passed | | +1 :green_heart: | compile | 0m 58s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 34s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | | +0 :ok: | spotbugs | 3m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 3m 50s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 35s | the patch passed | | +1 :green_heart: | compile | 1m 1s | the patch passed | | +1 :green_heart: | javac | 1m 1s | the patch passed | | +1 :green_heart: | checkstyle | 1m 33s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 4m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 16m 57s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 35s | the patch passed | | +1 :green_heart: | findbugs | 3m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 155m 55s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. | | | | 216m 10s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1015/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1015 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux d222a57dc4c4 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1015/out/precommit/personality/provided.sh | | git revision | branch-2 / 1217d57492 | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1015/2/testReport/ | | Max. process+thread count | 4606 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1015/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-23674) Too many rit page Numbers show confusion
dingbaosheng created HBASE-23674: Summary: Too many rit page Numbers show confusion Key: HBASE-23674 URL: https://issues.apache.org/jira/browse/HBASE-23674 Project: HBase Issue Type: Bug Components: master Affects Versions: 3.0.0 Reporter: dingbaosheng Fix For: 3.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] mygood opened a new pull request #1016: HBASE-23656 [MERGETOOL] HBASE Support Merge region by pattern
mygood opened a new pull request #1016: HBASE-23656 [MERGETOOL] HBASE Support Merge region by pattern URL: https://github.com/apache/hbase/pull/1016 Design Objective: 1. Merge empty region 2. Neat region 3. merge expired region Usage: bin/hbase onlinemerge [--tableName=] [--startRegion=] [--stopRegion=] [--maxRegionSize=] [--maxRegionCreateTime=] [--numMaxMergePlans=] [--targetRegionCount=] [--printExecutionPlan=] [--configMergePauseTime=] Options: --h or --h print help --tableName table name must be not null --startRegion start region --stopRegion stop region --maxRegionSize max region size Unit GB --maxRegionCreateTime max Region Create Time /MM/dd HH:mm:ss --numMaxMergePlans num MaxMerge Plans --targetRegionCount target Region Count --configMergePauseTime config Merge Pause Time In milliseconds --printExecutionPlan Value default is true print execution plans false is execution merge Examples: bin/hbase onlinemerge --tableName=test:test1 --startRegion=test:test1,,1576835912332.01d0d6c2b41e204104524d9aec6074fb. --stopRegion=test:test1,,1573044786980.0c9b5bd93f3b19eb9bd1a1011ddff66f. --maxRegionSize=0 --maxRegionCreateTime=/MM/dd HH:mm:ss --numMaxMergePlans=2 --targetRegionCount=4 --printExecutionPlan=false This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23656) [MERGETOOL] HBASE Support Merge region by pattern
[ https://issues.apache.org/jira/browse/HBASE-23656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-23656: - Attachment: HBASE-23656.branch-1.4.v1.patch > [MERGETOOL] HBASE Support Merge region by pattern > - > > Key: HBASE-23656 > URL: https://issues.apache.org/jira/browse/HBASE-23656 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-23656.branch-1.4.v1.patch, > HBASE-23656.master.v1.patch > > > Design Objective: > # Merge empty region > # Neat region > # merge expired region > Usage: bin/hbase onlinemerge [--tableName=] [--startRegion=] [--stopRegion=] > [--maxRegionSize=] [--maxRegionCreateTime=] [--numMaxMergePlans=] > [--targetRegionCount=] [--printExecutionPlan=] [--configMergePauseTime=] > Options: > --h or --h print help > --tableName table name must be not null > --startRegion start region > --stopRegion stop region > --maxRegionSize max region size Unit GB > --maxRegionCreateTime max Region Create Time /MM/dd HH:mm:ss > --numMaxMergePlans num MaxMerge Plans > --targetRegionCount target Region Count > --configMergePauseTime config Merge Pause Time In milliseconds > --printExecutionPlan Value default is true print execution plans false is > execution merge > Examples: > bin/hbase onlinemerge --tableName=test:test1 > --startRegion=test:test1,,1576835912332.01d0d6c2b41e204104524d9aec6074fb. > --stopRegion=test:test1,,1573044786980.0c9b5bd93f3b19eb9bd1a1011ddff66f. > --maxRegionSize=0 --maxRegionCreateTime=/MM/dd HH:mm:ss > --numMaxMergePlans=2 --targetRegionCount=4 --printExecutionPlan=false -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23656) [MERGETOOL] HBASE Support Merge region by pattern
[ https://issues.apache.org/jira/browse/HBASE-23656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengsicheng updated HBASE-23656: - Fix Version/s: 1.4.13 > [MERGETOOL] HBASE Support Merge region by pattern > - > > Key: HBASE-23656 > URL: https://issues.apache.org/jira/browse/HBASE-23656 > Project: HBase > Issue Type: New Feature > Components: master >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Minor > Fix For: 3.0.0, 1.4.13 > > Attachments: HBASE-23656.branch-1.4.v1.patch, > HBASE-23656.master.v1.patch > > > Design Objective: > # Merge empty region > # Neat region > # merge expired region > Usage: bin/hbase onlinemerge [--tableName=] [--startRegion=] [--stopRegion=] > [--maxRegionSize=] [--maxRegionCreateTime=] [--numMaxMergePlans=] > [--targetRegionCount=] [--printExecutionPlan=] [--configMergePauseTime=] > Options: > --h or --h print help > --tableName table name must be not null > --startRegion start region > --stopRegion stop region > --maxRegionSize max region size Unit GB > --maxRegionCreateTime max Region Create Time /MM/dd HH:mm:ss > --numMaxMergePlans num MaxMerge Plans > --targetRegionCount target Region Count > --configMergePauseTime config Merge Pause Time In milliseconds > --printExecutionPlan Value default is true print execution plans false is > execution merge > Examples: > bin/hbase onlinemerge --tableName=test:test1 > --startRegion=test:test1,,1576835912332.01d0d6c2b41e204104524d9aec6074fb. > --stopRegion=test:test1,,1573044786980.0c9b5bd93f3b19eb9bd1a1011ddff66f. > --maxRegionSize=0 --maxRegionCreateTime=/MM/dd HH:mm:ss > --numMaxMergePlans=2 --targetRegionCount=4 --printExecutionPlan=false -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23591) Negative memStoreSizing
[ https://issues.apache.org/jira/browse/HBASE-23591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012853#comment-17012853 ] Szabolcs Bukros commented on HBASE-23591: - [~anoop.hbase] You are right, we would loose the file ref... Thanks for pointing it out! What do you think about a solution where we send the last x (maybe 10?) HStoreFile paths in the FlushDescriptor instead of just the latest one? In the StoreFileManager the files are ordered by seqID so grabbing the last few is easy. We would have to make sure to filter out storedFiles already listed in the store of the replica region in replayFlush too. This way the first successful flush would add all the potentially missing refs. > Negative memStoreSizing > --- > > Key: HBASE-23591 > URL: https://issues.apache.org/jira/browse/HBASE-23591 > Project: HBase > Issue Type: Bug > Components: read replicas >Reporter: Szabolcs Bukros >Priority: Major > Fix For: 2.2.2 > > > After a flush on the replica region the memStoreSizing becomes negative: > {code:java} > 2019-12-17 08:31:59,983 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > 0beaae111b0f6e98bfde31ba35be5408 : Replaying flush marker action: > COMMIT_FLUSH table_name: "IntegrationTestRegionReplicaReplicati > on" encoded_region_name: "544affde3e027454f67c8ea46c8f69ee" > flush_sequence_number: 41392 store_flushes { family_name: "f1" > store_home_dir: "f1" flush_output: "3c48a23eac784a348a18e10e337d80a2" } > store_flushes { family_name: "f2" store_home_dir: "f2" flush_output: > "9a5283ec95694667b4ead2398af5f01e" } store_flushes { family_name: "f3" > store_home_dir: "f3" flush_output: "e6f25e6b0eca4d22af15d0626d0f8759" } > region_name: > "IntegrationTestRegionReplicaReplication,,1576599911697.544affde3e027454f67c8ea46c8f69ee." > 2019-12-17 08:31:59,984 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > 0beaae111b0f6e98bfde31ba35be5408 : Received a flush commit marker with > seqId:41392 and a previous prepared snapshot was found > 2019-12-17 08:31:59,993 INFO org.apache.hadoop.hbase.regionserver.HStore: > Region: 0beaae111b0f6e98bfde31ba35be5408 added > hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f1/3c48a23eac784a348a18e10e337d80a2, > entries=32445, sequenceid=41392, filesize=27.6 M > 2019-12-17 08:32:00,016 INFO org.apache.hadoop.hbase.regionserver.HStore: > Region: 0beaae111b0f6e98bfde31ba35be5408 added > hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f2/9a5283ec95694667b4ead2398af5f01e, > entries=12264, sequenceid=41392, filesize=10.9 M > 2019-12-17 08:32:00,121 INFO org.apache.hadoop.hbase.regionserver.HStore: > Region: 0beaae111b0f6e98bfde31ba35be5408 added > hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f3/e6f25e6b0eca4d22af15d0626d0f8759, > entries=32379, sequenceid=41392, filesize=27.5 M > 2019-12-17 08:32:00,122 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > CustomLog decrMemStoreSize. Current: dataSize=135810071, > getHeapSize=148400960, getOffHeapSize=0, getCellsCount=167243 delta: > dataSizeDelta=155923644, heapSizeDelta=170112320, offHeapSizeDelta=0, > cellsCountDelta=188399 > 2019-12-17 08:32:00,122 ERROR org.apache.hadoop.hbase.regionserver.HRegion: > Asked to modify this region's > (IntegrationTestRegionReplicaReplication,,1576599911697_0001.0beaae111b0f6e98bfde31ba35be54 > 08.) memStoreSizing to a negative value which is incorrect. Current > memStoreSizing=135810071, delta=-155923644 > java.lang.Exception > at > org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1323) > at > org.apache.hadoop.hbase.regionserver.HRegion.decrMemStoreSize(HRegion.java:1316) > at > org.apache.hadoop.hbase.regionserver.HRegion.decrMemStoreSize(HRegion.java:1303) > at > org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:5194) > at > org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:5025) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:1143) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:2232) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29754) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) > at > org.apache.hadoop.hb
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365221342 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java ## @@ -58,8 +62,19 @@ private final ZNodePaths znodePaths; + /** + * A znode maintained by MirroringTableStateManager. + * MirroringTableStateManager is deprecated to be removed in hbase3. It can also be disabled. + * Make sure it is enabled if you want to alter hbase:meta table in hbase2. In hbase3, + * TBD how metatable state will be hosted; likely on active hbase master. Review comment: So I do not think we should name this as mirrored, it is the original data. The state in master's memory, is a cache, actually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365230905 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseMetaEdit.java ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; +import org.apache.hadoop.hbase.client.RegionInfoBuilder; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.regionserver.Region; +import org.apache.hadoop.hbase.testclassification.LargeTests; +import org.apache.hadoop.hbase.testclassification.MiscTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.Before; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.rules.TestName; + +/** + * Test being able to edit hbase:meta. + */ +@Category({MiscTests.class, LargeTests.class}) +public class TestHBaseMetaEdit { + @ClassRule + public static final HBaseClassTestRule CLASS_RULE = + HBaseClassTestRule.forClass(TestHBaseMetaEdit.class); + @Rule + public TestName name = new TestName(); + private final static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @Before + public void before() throws Exception { +UTIL.startMiniCluster(); + } + + @After + public void after() throws Exception { +UTIL.shutdownMiniCluster(); + } + + /** + * Set versions, set HBASE-16213 indexed block encoding, and add a column family. + * Verify they are all in place by looking at TableDescriptor AND by checking + * what the RegionServer sees after opening Region. + */ + @Test + public void testEditMeta() throws IOException { +Admin admin = UTIL.getAdmin(); +admin.tableExists(TableName.META_TABLE_NAME); +admin.disableTable(TableName.META_TABLE_NAME); +assertTrue(admin.isTableDisabled(TableName.META_TABLE_NAME)); +TableDescriptor descriptor = admin.getDescriptor(TableName.META_TABLE_NAME); +ColumnFamilyDescriptor cfd = descriptor.getColumnFamily(HConstants.CATALOG_FAMILY); +byte [] extraColumnFamilyName = Bytes.toBytes("xtra"); +ColumnFamilyDescriptor newCfd = + ColumnFamilyDescriptorBuilder.newBuilder(extraColumnFamilyName).build(); +int oldVersions = cfd.getMaxVersions(); +// Add '1' to current versions count. +cfd = ColumnFamilyDescriptorBuilder.newBuilder(cfd).setMaxVersions(oldVersions + 1). +setConfiguration(ColumnFamilyDescriptorBuilder.DATA_BLOCK_ENCODING, +DataBlockEncoding.ROW_INDEX_V1.toString()).build(); +admin.modifyColumnFamily(TableName.META_TABLE_NAME, cfd); +admin.addColumnFamily(TableName.META_TABLE_NAME, newCfd); Review comment: Now we allow users to add/remove families of meta table from client side? A bit dangerous. I think we should only allow a sub set of alter operations for meta table and system tables from client side? At least, we should not let them add or remove a column family, it will introduce very critical problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365221090 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java ## @@ -58,8 +62,19 @@ private final ZNodePaths znodePaths; + /** + * A znode maintained by MirroringTableStateManager. + * MirroringTableStateManager is deprecated to be removed in hbase3. It can also be disabled. + * Make sure it is enabled if you want to alter hbase:meta table in hbase2. In hbase3, + * TBD how metatable state will be hosted; likely on active hbase master. Review comment: I think the state will either be on zk or on hdfs? HMaster itself is state less... In general, in the current architecture, I think it the state should be placed on zk, of course you could cache it in master and let client go to master to ask for the state. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365233018 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java ## @@ -25,51 +25,39 @@ /** * Get, remove and modify table descriptors. - * Used by servers to host descriptors. */ @InterfaceAudience.Private public interface TableDescriptors { /** - * @param tableName * @return TableDescriptor for tablename - * @throws IOException */ - TableDescriptor get(final TableName tableName) - throws IOException; + TableDescriptor get(final TableName tableName) throws IOException; /** * Get Map of all NamespaceDescriptors for a given namespace. * @return Map of all descriptors. - * @throws IOException */ - Map getByNamespace(String name) - throws IOException; + Map getByNamespace(String name) throws IOException; /** * Get Map of all TableDescriptors. Populates the descriptor cache as a * side effect. * Notice: the key of map is the table name which contains namespace. It was generated by * {@link TableName#getNameWithNamespaceInclAsString()}. * @return Map of all descriptors. - * @throws IOException */ Map getAll() throws IOException; /** * Add or update descriptor * @param htd Descriptor to set into TableDescriptors - * @throws IOException */ - void add(final TableDescriptor htd) - throws IOException; + void add(final TableDescriptor htd) throws IOException; Review comment: This method is a bit odd. We do not call it in CreateTableProcedure, but only in ModifyTableProcedure. Why not just call it update? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365227172 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java ## @@ -1,4 +1,6 @@ -/** +/* + * Copyright The Apache Software Foundation Review comment: What's this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365220125 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java ## @@ -666,42 +667,55 @@ public void run(PRESP resp) { new DisableTableProcedureBiConsumer(tableName)); } + /** + * Utility for completing passed TableState {@link CompletableFuture} future + * using passed parameters. + */ + private static CompletableFuture completeCheckTableState( Review comment: What is the return value used for? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365233838 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java ## @@ -135,16 +136,32 @@ public FSTableDescriptors(final Configuration conf, final FileSystem fs, * TODO: This is a workaround. Should remove this ugly code... */ public FSTableDescriptors(final Configuration conf, final FileSystem fs, -final Path rootdir, final boolean fsreadonly, final boolean usecache, -Function metaObserver) throws IOException { + final Path rootdir, final boolean fsreadonly, final boolean usecache, + Function metaObserver) throws IOException { this.fs = fs; this.rootdir = rootdir; this.fsreadonly = fsreadonly; this.usecache = usecache; -this.metaTableDescriptor = metaObserver == null ? createMetaTableDescriptor(conf) - : metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build(); +TableDescriptor td = null; +try { + td = getTableDescriptorFromFs(fs, rootdir, TableName.META_TABLE_NAME); +} catch (TableInfoMissingException e) { + td = metaObserver == null? createMetaTableDescriptor(conf): +metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build(); + if (!fsreadonly) { +LOG.info("Creating new hbase:meta table default descriptor/schema {}", td); +updateTableDescriptor(td); + } +} +this.metaTableDescriptor = td; } + /** + * + * Should be private + * @deprecated Since 2.3.0. Should be for internal use only. Used by testing. Review comment: I do not think this should be deprecated? The class is IA.Private so it is OK to put internal methods here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365224489 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java ## @@ -229,6 +244,43 @@ private void getMetaRegionLocation(CompletableFuture future, }); } + @Override + public CompletableFuture getMetaTableState() { +return getAndConvert(this.znodeMirroredMetaTableState, ZKAsyncRegistry::getTableState). + thenApply(state -> { +return state == null || state.equals(ENABLED_META_TABLE_STATE.getState())? + ENABLED_META_TABLE_STATE: new TableState(TableName.META_TABLE_NAME, state); + }).exceptionally(e -> { Review comment: Currently in HBase, usually we will create a new CompletableFuture and use FutureUtils.addListener to complete it. The code in exceptionally are a bit tricky, where we throw a CompletionException... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
Apache9 commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365226730 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java ## @@ -54,8 +53,20 @@ // TODO: Make this a guava Service @InterfaceAudience.Private public class TableStateManager { - private static final Logger LOG = LoggerFactory.getLogger(TableStateManager.class); + + /** + * All table state is kept in hbase:meta except that of hbase:meta itself. Review comment: In general I do not think this is a good design. We should persist it somewhere, and just cache it in master's memory. And we do not need to disable a table when altering any more? And the solution here just assume that we only have one meta region? So the altering operation can be atomic? What if we have multiple meta regions and we crash in the middle? I think this patch also aims to implement splittable meta in the future? No? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012871#comment-17012871 ] Duo Zhang commented on HBASE-23055: --- Replied on RB, I still do not think this is a good solution. We should design for the future, for splittable meta, not for a temporary solution. And I do not get the point why we jsut make the state for meta table volatile, why not just store it on zk? And when reviewing the patch, a good thing is that, we do not reference the meta table descriptor everywhere in the code. We just reference the region info everywhere, which I think it is OK for now as the region info does not include the table descriptor. This can be done in the future when implementing splittable meta. So I think we could get the patch in on master, or even on branch-2, but not the current one... Thanks. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23675) Move to Apache parent POM version 22
Peter Somogyi created HBASE-23675: - Summary: Move to Apache parent POM version 22 Key: HBASE-23675 URL: https://issues.apache.org/jira/browse/HBASE-23675 Project: HBase Issue Type: Improvement Components: dependencies Affects Versions: 3.0.0, 2.3.0 Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: 3.0.0, 2.3.0 Apache parent POM version 22 was released on 2020/01/09. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23663) Allow dot and hyphen in Profiler's URL
[ https://issues.apache.org/jira/browse/HBASE-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012884#comment-17012884 ] Hudson commented on HBASE-23663: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Allow dot and hyphen in Profiler's URL > -- > > Key: HBASE-23663 > URL: https://issues.apache.org/jira/browse/HBASE-23663 > Project: HBase > Issue Type: Bug > Components: profiler >Affects Versions: 3.0.0, 2.2.0, 2.3.0, 2.1.5 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > ProfileOutputServlet sanitizes the input URL but fails when the query string > contains dot or hyphen. These are valid characters for the hostname. > Example URL part: > /prof-output/async-prof-pid-122466-cpu-1.svg?host=myhost-1.example.com&port=16010 > > In this case the user gets the following error message: > {noformat} > HTTP ERROR: 500 > Problem accessing /prof-output/async-prof-pid-122466-cpu-1.svg. Reason: > java.lang.RuntimeException: Non-alphanumeric data found in input, > aborting. > Powered by Jetty:// 9.3.27.v20190418 {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012885#comment-17012885 ] Hudson commented on HBASE-23655: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23654) Please add Apache Trafodion and EsgynDB to "Powered by Apache HBase" page
[ https://issues.apache.org/jira/browse/HBASE-23654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012886#comment-17012886 ] Hudson commented on HBASE-23654: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Please add Apache Trafodion and EsgynDB to "Powered by Apache HBase" page > - > > Key: HBASE-23654 > URL: https://issues.apache.org/jira/browse/HBASE-23654 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Dave Birdsall >Assignee: Beata Sudi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9 > > > Please add the following two entries to the web page "Powered By Apache > HBase™" ([https://hbase.apache.org/poweredbyhbase.html]): > Trafodion (link [https://trafodion.apache.org/]) > Apache Trafodion(TM) is a webscale SQL-on-Hadoop solution enabling > transactional or operational workloads. It uses HBase as its storage engine > for SQL tables. > EsgynDB (link [https://esgyn.com/]) > EsgynDB, powered by Apache Trafodion(TM), provides enterprise SQL on Hadoop. > It includes full ACID transactions, online transaction processing and online > analytic processing, along with enterprise features such as disaster recovery > and full backup/restore. Native tables are stored in HBase, but read and > write access to various other file formats such as Apache Parquet and ORC is > also supported. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23664) Upgrade JUnit to 4.13
[ https://issues.apache.org/jira/browse/HBASE-23664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012887#comment-17012887 ] Hudson commented on HBASE-23664: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Upgrade JUnit to 4.13 > - > > Key: HBASE-23664 > URL: https://issues.apache.org/jira/browse/HBASE-23664 > Project: HBase > Issue Type: Task > Components: integration tests, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > New JUnit released a week ago. Let's give it a spin. > https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.13.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23378) Clean Up FSUtil setClusterId
[ https://issues.apache.org/jira/browse/HBASE-23378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012889#comment-17012889 ] Hudson commented on HBASE-23378: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Clean Up FSUtil setClusterId > > > Key: HBASE-23378 > URL: https://issues.apache.org/jira/browse/HBASE-23378 > Project: HBase > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 3.0.0 > > > * Use try-with-resources > * Remove bad practice of catching one's own Exceptions > * Method signature 'wait' should be of type long to match JDK API > * Add additional debugging > * Do not swallow Interrupt status of thread > * General cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22285) A normalizer which merges very small size regions with adjacent regions.(MergeToNormalize)
[ https://issues.apache.org/jira/browse/HBASE-22285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012888#comment-17012888 ] Hudson commented on HBASE-22285: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > A normalizer which merges very small size regions with adjacent > regions.(MergeToNormalize) > -- > > Key: HBASE-22285 > URL: https://issues.apache.org/jira/browse/HBASE-22285 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Aman Poonia >Assignee: Aman Poonia >Priority: Minor > Fix For: 3.0.0 > > > There are scenarios where we have seen around 5% of total regions with a size > of 0 bytes and another 5-6 % regions with size in a few bytes. These kinds of > regions increase with time considering we have TTL over the rows. > After exploring the option of RegionNormalizer and doing some quick runs we > found that that is not suitable considering it also splits the regions and > merges to normalize. What we really want is to split as per Split policy and > merge very small regions with adjacent regions to make sure we reduce 0-byte > regions. > We can plugin this normalizer using the property > "hbase.master.normalizer.class" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012890#comment-17012890 ] Hudson commented on HBASE-23055: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012891#comment-17012891 ] Hudson commented on HBASE-23659: Results for branch master [build #1594 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1594/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/master/1594//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012892#comment-17012892 ] Duo Zhang commented on HBASE-23055: --- Or let's open a follow on issue, and set it as a blocker of 2.3.0, to finish the altering meta change, but persisting the state of meta table, and implement alter the alter meta table procedure. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #1017: HBASE-23675 Move to Apache parent POM version 22
petersomogyi opened a new pull request #1017: HBASE-23675 Move to Apache parent POM version 22 URL: https://github.com/apache/hbase/pull/1017 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (HBASE-23675) Move to Apache parent POM version 22
[ https://issues.apache.org/jira/browse/HBASE-23675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23675 started by Peter Somogyi. - > Move to Apache parent POM version 22 > > > Key: HBASE-23675 > URL: https://issues.apache.org/jira/browse/HBASE-23675 > Project: HBase > Issue Type: Improvement > Components: dependencies >Affects Versions: 3.0.0, 2.3.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Apache parent POM version 22 was released on 2020/01/09. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23652) Move the unsupported procedure type check before migrating to RegionProcedureStore
[ https://issues.apache.org/jira/browse/HBASE-23652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23652: -- Summary: Move the unsupported procedure type check before migrating to RegionProcedureStore (was: Sort out how to handle v2.1 procedures during upgrade) > Move the unsupported procedure type check before migrating to > RegionProcedureStore > -- > > Key: HBASE-23652 > URL: https://issues.apache.org/jira/browse/HBASE-23652 > Project: HBase > Issue Type: Task > Components: master >Affects Versions: 2.2.2 >Reporter: Nick Dimiduk >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.3.0 > > > Per > [discussion|https://lists.apache.org/thread.html/c5f960e6cf3c89ffbf8f8302afb121462a08fd9a89d384c6795a7b81%40%3Cdev.hbase.apache.org%3E] > on the list, let's figure out how to make the upgrade from a v2.1 procedure > store less error-prone. Could be a simple as documenting runbook steps to > execute during the rolling upgrade, but it would be nice if the software > could roll over the data versions gracefully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 opened a new pull request #1018: HBASE-23652 Move the unsupported procedure type check before migratin…
Apache9 opened a new pull request #1018: HBASE-23652 Move the unsupported procedure type check before migratin… URL: https://github.com/apache/hbase/pull/1018 …g to RegionProcedureStore This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[ https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012947#comment-17012947 ] Hudson commented on HBASE-23601: Results for branch branch-2.1 [build #1768 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1768/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1768//General_Nightly_Build_Report/] (x) {color:red}-1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1768//JDK8_Nightly_Build_Report_(Hadoop2)/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1768//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > OutputSink.WriterThread exception gets stuck and repeated indefinietly > -- > > Key: HBASE-23601 > URL: https://issues.apache.org/jira/browse/HBASE-23601 > Project: HBase > Issue Type: Bug > Components: read replicas >Affects Versions: 2.2.2 >Reporter: Szabolcs Bukros >Assignee: Szabolcs Bukros >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4 > > > When a WriterThread runs into an exception (ie: NotServingRegionException), > the exception is stored in the controller. It is never removed and can not be > overwritten either. > > {code:java} > public void run() { > try { > doRun(); > } catch (Throwable t) { > LOG.error("Exiting thread", t); > controller.writerThreadError(t); > } > }{code} > Thanks to this every time PipelineController.checkForErrors() is called the > same old exception is rethrown. > > For example in RegionReplicaReplicationEndpoint.replicate there is a while > loop that does the actual replicating. Every time it loops, it calls > checkForErrors(), catches the rethrown exception, logs it but does nothing > about it. This results in ~2GB log files in ~5min in my experience. > > My proposal would be to clean up the stored exception when it reaches > RegionReplicaReplicationEndpoint.replicate and make sure we restart the > WriterThread that died throwing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk merged pull request #1009: HBASE-23664 Upgrade JUnit to 4.13
ndimiduk merged pull request #1009: HBASE-23664 Upgrade JUnit to 4.13 URL: https://github.com/apache/hbase/pull/1009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13
ndimiduk commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13 URL: https://github.com/apache/hbase/pull/1010#issuecomment-573078633 That's a lot of failures. Kicking the PR job to try again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365324010 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java ## @@ -666,42 +667,55 @@ public void run(PRESP resp) { new DisableTableProcedureBiConsumer(tableName)); } + /** + * Utility for completing passed TableState {@link CompletableFuture} future + * using passed parameters. + */ + private static CompletableFuture completeCheckTableState( Review comment: Looks like this a holdover. I can make it void. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365330389 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java ## @@ -58,8 +62,19 @@ private final ZNodePaths znodePaths; + /** + * A znode maintained by MirroringTableStateManager. + * MirroringTableStateManager is deprecated to be removed in hbase3. It can also be disabled. + * Make sure it is enabled if you want to alter hbase:meta table in hbase2. In hbase3, + * TBD how metatable state will be hosted; likely on active hbase master. Review comment: The 'mirroring' feature predates this patch. This is an old facility this patch just makes use of. Changing this old features name is outside realm of this work. You get the bit that edit of meta is a rare, short-lived event and you get the bit that this implementation fails the state back to ENABLED if ever an issue, intentionally, to minimize our running into exotic situations. I don't want a permanently DISABLED state for hbase:meta, at least not at this stage in the game. We can do that later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365331618 ## File path: hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java ## @@ -229,6 +244,43 @@ private void getMetaRegionLocation(CompletableFuture future, }); } + @Override + public CompletableFuture getMetaTableState() { +return getAndConvert(this.znodeMirroredMetaTableState, ZKAsyncRegistry::getTableState). + thenApply(state -> { +return state == null || state.equals(ENABLED_META_TABLE_STATE.getState())? + ENABLED_META_TABLE_STATE: new TableState(TableName.META_TABLE_NAME, state); + }).exceptionally(e -> { Review comment: Yeah, saw that. Here, if no znode -- probably because the mirroring table state manager is turned off -- then I want the return to be ENABLED (Completed CF). Should I change this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365337029 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java ## @@ -54,8 +53,20 @@ // TODO: Make this a guava Service @InterfaceAudience.Private public class TableStateManager { - private static final Logger LOG = LoggerFactory.getLogger(TableStateManager.class); + + /** + * All table state is kept in hbase:meta except that of hbase:meta itself. Review comment: bq. We should persist it somewhere, and just cache it in master's memory. I explain why state is intentionally transient for now so table falls-back to ENABLED if problem like Master crash during alter; its one less thing for the operator to decipher and hbck2 navigate starting up a cluster; in other words we fallback to the 'known' state rather than persist DISABLED meta, a new condition that is sure to have holes in it. bq. And the solution here just assume that we only have one meta region? All of the code base presumes this, not just this patch (this patch is part work to undo this presumption letting go of hardcoded meta schema). bq. I think this patch also aims to implement splittable meta in the future? No? Splittable meta is another project, not this one. bq. And we do not need to disable a table when altering any more? This patch makes hbase:meta dynamic. Maybe the above will work. Let me try. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365337458 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseMetaEdit.java ## @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; +import org.apache.hadoop.hbase.client.RegionInfoBuilder; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding; +import org.apache.hadoop.hbase.regionserver.Region; +import org.apache.hadoop.hbase.testclassification.LargeTests; +import org.apache.hadoop.hbase.testclassification.MiscTests; +import org.apache.hadoop.hbase.util.Bytes; +import org.junit.After; +import org.junit.Before; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.rules.TestName; + +/** + * Test being able to edit hbase:meta. + */ +@Category({MiscTests.class, LargeTests.class}) +public class TestHBaseMetaEdit { + @ClassRule + public static final HBaseClassTestRule CLASS_RULE = + HBaseClassTestRule.forClass(TestHBaseMetaEdit.class); + @Rule + public TestName name = new TestName(); + private final static HBaseTestingUtility UTIL = new HBaseTestingUtility(); + + @Before + public void before() throws Exception { +UTIL.startMiniCluster(); + } + + @After + public void after() throws Exception { +UTIL.shutdownMiniCluster(); + } + + /** + * Set versions, set HBASE-16213 indexed block encoding, and add a column family. + * Verify they are all in place by looking at TableDescriptor AND by checking + * what the RegionServer sees after opening Region. + */ + @Test + public void testEditMeta() throws IOException { +Admin admin = UTIL.getAdmin(); +admin.tableExists(TableName.META_TABLE_NAME); +admin.disableTable(TableName.META_TABLE_NAME); +assertTrue(admin.isTableDisabled(TableName.META_TABLE_NAME)); +TableDescriptor descriptor = admin.getDescriptor(TableName.META_TABLE_NAME); +ColumnFamilyDescriptor cfd = descriptor.getColumnFamily(HConstants.CATALOG_FAMILY); +byte [] extraColumnFamilyName = Bytes.toBytes("xtra"); +ColumnFamilyDescriptor newCfd = + ColumnFamilyDescriptorBuilder.newBuilder(extraColumnFamilyName).build(); +int oldVersions = cfd.getMaxVersions(); +// Add '1' to current versions count. +cfd = ColumnFamilyDescriptorBuilder.newBuilder(cfd).setMaxVersions(oldVersions + 1). +setConfiguration(ColumnFamilyDescriptorBuilder.DATA_BLOCK_ENCODING, +DataBlockEncoding.ROW_INDEX_V1.toString()).build(); +admin.modifyColumnFamily(TableName.META_TABLE_NAME, cfd); +admin.addColumnFamily(TableName.META_TABLE_NAME, newCfd); Review comment: This is a good point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365337598 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java ## @@ -25,51 +25,39 @@ /** * Get, remove and modify table descriptors. - * Used by servers to host descriptors. */ @InterfaceAudience.Private public interface TableDescriptors { /** - * @param tableName * @return TableDescriptor for tablename - * @throws IOException */ - TableDescriptor get(final TableName tableName) - throws IOException; + TableDescriptor get(final TableName tableName) throws IOException; /** * Get Map of all NamespaceDescriptors for a given namespace. * @return Map of all descriptors. - * @throws IOException */ - Map getByNamespace(String name) - throws IOException; + Map getByNamespace(String name) throws IOException; /** * Get Map of all TableDescriptors. Populates the descriptor cache as a * side effect. * Notice: the key of map is the table name which contains namespace. It was generated by * {@link TableName#getNameWithNamespaceInclAsString()}. * @return Map of all descriptors. - * @throws IOException */ Map getAll() throws IOException; /** * Add or update descriptor * @param htd Descriptor to set into TableDescriptors - * @throws IOException */ - void add(final TableDescriptor htd) - throws IOException; + void add(final TableDescriptor htd) throws IOException; Review comment: Let me fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365338167 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java ## @@ -135,16 +136,32 @@ public FSTableDescriptors(final Configuration conf, final FileSystem fs, * TODO: This is a workaround. Should remove this ugly code... */ public FSTableDescriptors(final Configuration conf, final FileSystem fs, -final Path rootdir, final boolean fsreadonly, final boolean usecache, -Function metaObserver) throws IOException { + final Path rootdir, final boolean fsreadonly, final boolean usecache, + Function metaObserver) throws IOException { this.fs = fs; this.rootdir = rootdir; this.fsreadonly = fsreadonly; this.usecache = usecache; -this.metaTableDescriptor = metaObserver == null ? createMetaTableDescriptor(conf) - : metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build(); +TableDescriptor td = null; +try { + td = getTableDescriptorFromFs(fs, rootdir, TableName.META_TABLE_NAME); +} catch (TableInfoMissingException e) { + td = metaObserver == null? createMetaTableDescriptor(conf): +metaObserver.apply(createMetaTableDescriptorBuilder(conf)).build(); + if (!fsreadonly) { +LOG.info("Creating new hbase:meta table default descriptor/schema {}", td); +updateTableDescriptor(td); + } +} +this.metaTableDescriptor = td; } + /** + * + * Should be private + * @deprecated Since 2.3.0. Should be for internal use only. Used by testing. Review comment: You are right. I wanted to take it private/protected but as you note, its Private so I can just do it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta
saintstack commented on a change in pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#discussion_r365340349 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java ## @@ -1,4 +1,6 @@ -/** +/* + * Copyright The Apache Software Foundation Review comment: We've had this discussion before. Its not javadoc so seems right to remove the extra '*'. We should do up an agreement on way to go. Did you say why you think we should preserve the '**'? Is it because its in the formatter? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack opened a new pull request #1011: HBASE-23055 Alter hbase:meta
saintstack opened a new pull request #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011 Make it so hbase:meta can be altered. TableState for hbase:meta was hardcoded ENABLED. Make it dynamic. State is now kept in current active Master. It is transient so falls back to default if Master crashes. Add to registry a getMetaTableState which reads mirrored state from zookeeper (NOT from Master and defaults ENABLED if no implementation or error fetching state). hbase:meta schema will be bootstrapped from the filesystem. Changes to filesystem schema are atomic so we should be ok if Master fails mid-edit (TBD). Undoes a bunch of guards that prevented our being able to edit hbase:meta. TODO: Tests, more clarity around hbase:meta table state, and undoing references to hard-coded hbase:meta regioninfo. M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java Throw illegal access exception if you try to use MetaTableAccessor getting state of the hbase:meta table. M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java Add fetching of hbase:meta table state from registry. Adds cache of tablestates w/ a ttl of 1 second (adjustable). M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java Add querying registry for hbase:meta table state. M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java Add querying of mirrored table state for hbase:meta table. M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java Shutdown access. M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java Just cleanup. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java Add state holder for hbase:meta. Removed unused methods. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Shut down access. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java Allow hbase:meta to be disabled. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java Allow hbase:meta to be enabled. Signed-off-by: Bharath Vissapragada This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1011: HBASE-23055 Alter hbase:meta
saintstack commented on issue #1011: HBASE-23055 Alter hbase:meta URL: https://github.com/apache/hbase/pull/1011#issuecomment-573125703 @Apache9 Thanks for the review. Let me open a follow-on to address helpful feedback (Boolean => Void, Preventing CF deletion, etc.). On the 'bad design', your objections seem a little out-of-place ('... presumes single meta region...') or easy to counter and your worries that this patch unsettles the codebase seem to pale compared to what has already gone into this minor branch. How to proceed? A discussion on dev list? Or would a write up on how this works help? A one-pager? Let me know. Let me reopen this while we are commenting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013079#comment-17013079 ] Sean Busbey commented on HBASE-23055: - I agree that this change looks much lower risk than the procedure store refactoring. (I also want both of them on branch-2 if they're not in feature branches.) > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23055) Alter hbase:meta
[ https://issues.apache.org/jira/browse/HBASE-23055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013083#comment-17013083 ] Sean Busbey commented on HBASE-23055: - bq. Or let's open a follow on issue, and set it as a blocker of 2.3.0, to finish the altering meta change, but persisting the state of meta table, and implement alter the alter meta table procedure. we're overdue on the 2.3.z release line starting. an issue for dev@, but maybe we should start RCs off an earlier point in the branch-2 history. > Alter hbase:meta > > > Key: HBASE-23055 > URL: https://issues.apache.org/jira/browse/HBASE-23055 > Project: HBase > Issue Type: Task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > hbase:meta is currently hardcoded. Its schema cannot be change. > This issue is about allowing edits to hbase:meta schema. It will allow our > being able to set encodings such as the block-with-indexes which will help > quell CPU usage on host carrying hbase:meta. A dynamic hbase:meta is first > step on road to being able to split meta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23676) Address feedback on HBASE-23055 Alter hbase:meta.
Michael Stack created HBASE-23676: - Summary: Address feedback on HBASE-23055 Alter hbase:meta. Key: HBASE-23676 URL: https://issues.apache.org/jira/browse/HBASE-23676 Project: HBase Issue Type: Bug Affects Versions: 2.3.0 Reporter: Michael Stack Assignee: Michael Stack Fix For: 2.3.0 Good feedback on HBASE-23055 came in after merge from [~zhangduo]. Opening this issue to address it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1016: HBASE-23656 [MERGETOOL] HBASE Support Merge region by pattern
Apache-HBase commented on issue #1016: HBASE-23656 [MERGETOOL] HBASE Support Merge region by pattern URL: https://github.com/apache/hbase/pull/1016#issuecomment-573138221 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ branch-1.4 Compile Tests _ | | +0 :ok: | mvndep | 1m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 9m 0s | branch-1.4 passed | | +1 :green_heart: | compile | 3m 4s | branch-1.4 passed with JDK v1.8.0_232 | | +1 :green_heart: | compile | 2m 48s | branch-1.4 passed with JDK v1.7.0_242 | | +1 :green_heart: | checkstyle | 8m 22s | branch-1.4 passed | | +1 :green_heart: | shadedjars | 2m 45s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 35s | branch-1.4 passed with JDK v1.8.0_232 | | +1 :green_heart: | javadoc | 3m 37s | branch-1.4 passed with JDK v1.7.0_242 | | +0 :ok: | spotbugs | 2m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 13m 5s | branch-1.4 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 57s | the patch passed | | +1 :green_heart: | compile | 1m 43s | the patch passed with JDK v1.8.0_232 | | +1 :green_heart: | javac | 1m 43s | the patch passed | | +1 :green_heart: | compile | 1m 46s | the patch passed with JDK v1.7.0_242 | | +1 :green_heart: | javac | 1m 46s | the patch passed | | +1 :green_heart: | checkstyle | 6m 39s | the patch passed | | +1 :green_heart: | shellcheck | 0m 2s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 2m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 2m 32s | Patch does not cause any errors with Hadoop 2.7.7. | | +1 :green_heart: | javadoc | 2m 34s | the patch passed with JDK v1.8.0_232 | | +1 :green_heart: | javadoc | 3m 38s | the patch passed with JDK v1.7.0_242 | | +1 :green_heart: | findbugs | 13m 21s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 208m 46s | root in the patch passed. | | +1 :green_heart: | asflicense | 1m 9s | The patch does not generate ASF License warnings. | | | | 300m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1016/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1016 | | Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 709cf1c4a6d2 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1016/out/precommit/personality/provided.sh | | git revision | branch-1.4 / 879b7ea | | Default Java | 1.7.0_242 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1016/1/testReport/ | | Max. process+thread count | 4130 (vs. ulimit of 1) | | modules | C: hbase-server . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1016/1/console | | versions | git=1.9.1 maven=3.0.5 shellcheck=0.7.0 findbugs=3.0.1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Gi
[jira] [Created] (HBASE-23677) region.jsp returns 500/NPE when provided encoded region name is not online
Nick Dimiduk created HBASE-23677: Summary: region.jsp returns 500/NPE when provided encoded region name is not online Key: HBASE-23677 URL: https://issues.apache.org/jira/browse/HBASE-23677 Project: HBase Issue Type: Bug Components: regionserver, UI Affects Versions: 2.2.3 Reporter: Nick Dimiduk Noticed while testing for HBASE-23653. A link off to a region that is closed or otherwise not in the region server's online region map results in a 500. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1017: HBASE-23675 Move to Apache parent POM version 22
Apache-HBase commented on issue #1017: HBASE-23675 Move to Apache parent POM version 22 URL: https://github.com/apache/hbase/pull/1017#issuecomment-573145861 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 42s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 53s | master passed | | +1 :green_heart: | compile | 3m 17s | master passed | | +1 :green_heart: | shadedjars | 5m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 5m 32s | the patch passed | | +1 :green_heart: | compile | 3m 21s | the patch passed | | +1 :green_heart: | javac | 3m 21s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedjars | 5m 0s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 20s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 3m 4s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 179m 46s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 59s | The patch does not generate ASF License warnings. | | | | 241m 42s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.client.TestAsyncTableGetMultiThreaded | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1017/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1017 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 7d1c60a90af4 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1017/out/precommit/personality/provided.sh | | git revision | master / 56842d04d7 | | Default Java | 1.8.0_181 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1017/1/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1017/1/testReport/ | | Max. process+thread count | 5375 (vs. ulimit of 1) | | modules | C: hbase-build-configuration . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1017/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…
saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… URL: https://github.com/apache/hbase/pull/1012#issuecomment-573148773 So, the ROW_INDEX_V1 encoding seems to mess up the tests. Here is a patch with ROWCOL bloom to see how it does. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #1018: HBASE-23652 Move the unsupported procedure type check before migratin…
Apache-HBase commented on issue #1018: HBASE-23652 Move the unsupported procedure type check before migratin… URL: https://github.com/apache/hbase/pull/1018#issuecomment-573151752 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 9s | master passed | | +1 :green_heart: | compile | 0m 58s | master passed | | +1 :green_heart: | checkstyle | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 5m 4s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 39s | master passed | | +0 :ok: | spotbugs | 4m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 50s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 33s | the patch passed | | +1 :green_heart: | compile | 0m 58s | the patch passed | | +1 :green_heart: | javac | 0m 58s | the patch passed | | -1 :x: | checkstyle | 1m 27s | hbase-server: The patch generated 5 new + 103 unchanged - 0 fixed = 108 total (was 103) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 17m 45s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | | +1 :green_heart: | findbugs | 4m 57s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 162m 52s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 227m 14s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1018 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux b1b05b15086e 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1018/out/precommit/personality/provided.sh | | git revision | master / 56842d04d7 | | Default Java | 1.8.0_181 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/1/artifact/out/diff-checkstyle-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/1/testReport/ | | Max. process+thread count | 4879 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1018/1/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack closed pull request #1015: HBASE-23668 Master log start filling with "Flush journal status" messages
saintstack closed pull request #1015: HBASE-23668 Master log start filling with "Flush journal status" messages URL: https://github.com/apache/hbase/pull/1015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] saintstack commented on issue #1015: HBASE-23668 Master log start filling with "Flush journal status" messages
saintstack commented on issue #1015: HBASE-23668 Master log start filling with "Flush journal status" messages URL: https://github.com/apache/hbase/pull/1015#issuecomment-573152004 Manually merged to branch-2 and master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23668) Master log start filling with "Flush journal status" messages
[ https://issues.apache.org/jira/browse/HBASE-23668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-23668. --- Hadoop Flags: Reviewed Assignee: Michael Stack Resolution: Fixed Merged to branch-2+. Thanks for review [~zhangduo] (reopen to add your UT?). > Master log start filling with "Flush journal status" messages > - > > Key: HBASE-23668 > URL: https://issues.apache.org/jira/browse/HBASE-23668 > Project: HBase > Issue Type: Improvement > Components: proc-v2, RegionProcedureStore >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Takes a while to get into this condition. Not each to tell how because all > logs have rolled off and I only have logs filled w/ below: > {code} > 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush status journal: > Acquiring readlock on region at 1578553261723 > Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, > failureReason:Nothing to flush,flush seq id45226854 at 1578553261723 > 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush status journal: > Acquiring readlock on region at 1578553261723 > Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, > failureReason:Nothing to flush,flush seq id45226855 at 1578553261723 > 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush status journal: > Acquiring readlock on region at 1578553261723 > Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, > failureReason:Nothing to flush,flush seq id45226856 at 1578553261723 > 2020-01-09 07:01:01,723 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Flush status journal: > Acquiring readlock on region at 1578553261723 > Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY, > failureReason:Nothing to flush,flush seq id45226857 at 1578553261723 > {code} > ... I added the printing of flushresult... i.e. cannot flush because store is > empty. > Digging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13
Apache-HBase commented on issue #1010: HBASE-23664 Upgrade JUnit to 4.13 URL: https://github.com/apache/hbase/pull/1010#issuecomment-573157065 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 7m 11s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ branch-1 Compile Tests _ | | +1 :green_heart: | mvninstall | 8m 43s | branch-1 passed | | +1 :green_heart: | compile | 1m 42s | branch-1 passed with JDK v1.8.0_232 | | +1 :green_heart: | compile | 1m 49s | branch-1 passed with JDK v1.7.0_242 | | +1 :green_heart: | shadedjars | 2m 49s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 13s | branch-1 passed with JDK v1.8.0_232 | | +1 :green_heart: | javadoc | 3m 46s | branch-1 passed with JDK v1.7.0_242 | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 2m 1s | the patch passed | | +1 :green_heart: | compile | 1m 42s | the patch passed with JDK v1.8.0_232 | | +1 :green_heart: | javac | 1m 42s | the patch passed | | +1 :green_heart: | compile | 1m 45s | the patch passed with JDK v1.7.0_242 | | +1 :green_heart: | javac | 1m 45s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 1s | The patch has 1 ill-formed XML file(s). | | +1 :green_heart: | shadedjars | 2m 43s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 0s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | +1 :green_heart: | javadoc | 2m 7s | the patch passed with JDK v1.8.0_232 | | +1 :green_heart: | javadoc | 3m 48s | the patch passed with JDK v1.7.0_242 | ||| _ Other Tests _ | | -1 :x: | unit | 147m 38s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 198m 52s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | pom.xml | | Failed junit tests | hadoop.hbase.client.TestAdmin2 | | | hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplicas | | | hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory | | | hadoop.hbase.regionserver.TestCompactionState | | | hadoop.hbase.security.token.TestGenerateDelegationToken | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1010 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 84529f3a5ea7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1010/out/precommit/personality/provided.sh | | git revision | branch-1 / 6b940ae | | Default Java | 1.7.0_242 | | Multi-JDK versions | /usr/lib/jvm/zulu-8-amd64:1.8.0_232 /usr/lib/jvm/zulu-7-amd64:1.7.0_242 | | xml | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/xml.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/testReport/ | | Max. process+thread count | 4560 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1010/2/console | | versions | git=1.9.1 maven=3.0.5 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23279) Switch default block encoding to ROW_INDEX_V1
[ https://issues.apache.org/jira/browse/HBASE-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013140#comment-17013140 ] Michael Stack commented on HBASE-23279: --- How we doing here [~vjasani]? [~anoop.hbase] reminded me this was outstanding in his review of HBASE-21065. In my test runs enabling ROW_INDEX_V1 on hbase:meta, the TestGetClosestAtOrBefore fails (commented out here) and then the TestProcedurePriority fails too as it does for you here. What you thinking? I was going to dig in but asking here first. > Switch default block encoding to ROW_INDEX_V1 > - > > Key: HBASE-23279 > URL: https://issues.apache.org/jira/browse/HBASE-23279 > Project: HBase > Issue Type: Wish >Affects Versions: 3.0.0, 2.3.0 >Reporter: Lars Hofhansl >Assignee: Viraj Jasani >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23279.master.000.patch, > HBASE-23279.master.001.patch, HBASE-23279.master.002.patch, > HBASE-23279.master.003.patch > > > Currently we set both block encoding and compression to NONE. > ROW_INDEX_V1 has many advantages and (almost) no disadvantages (the hfiles > are slightly larger about 3% or so). I think that would a better default than > NONE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23678) Literate builder API for version management in schema
Andrew Kyle Purtell created HBASE-23678: --- Summary: Literate builder API for version management in schema Key: HBASE-23678 URL: https://issues.apache.org/jira/browse/HBASE-23678 Project: HBase Issue Type: Improvement Reporter: Andrew Kyle Purtell Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify _num_versions_ in this example since version 0.1. A literate builder API, by way if its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema
[ https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23678: Description: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way if its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. was: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify _num_versions_ in this example since version 0.1. A literate builder API, by way if its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. > Literate builder API for version management in schema > - > > Key: HBASE-23678 > URL: https://issues.apache.org/jira/browse/HBASE-23678 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and > KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance > regarding their usage. Almost all combinations of these four settings make > sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and > KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the > behavior with TTL easier to conceive when creating the schema. This could > take the form of a literate builder API for ColumnDescriptor or an extension > to an existing one. > Let me give you a motivating example: We may want to retain all versions for > a given TTL, and then only a specific number of versions. This can be > achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, > KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive > though because VERSIONS has been used to specify the number of versions to > retain (_num_versions_ in this example) since HBase version 0.1, so this is > going to be a source of confusion - I've seen it in practice. > A literate builder API, by way if its method names, could let a user describe > more or less in speaking language how they want version retention to work, > and internally the builder API could set the low level schema attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema
[ https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23678: Description: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. was: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way if its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. > Literate builder API for version management in schema > - > > Key: HBASE-23678 > URL: https://issues.apache.org/jira/browse/HBASE-23678 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and > KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance > regarding their usage. Almost all combinations of these four settings make > sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and > KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the > behavior with TTL easier to conceive when creating the schema. This could > take the form of a literate builder API for ColumnDescriptor or an extension > to an existing one. > Let me give you a motivating example: We may want to retain all versions for > a given TTL, and then only a specific number of versions. This can be > achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, > KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive > though because VERSIONS has been used to specify the number of versions to > retain (_num_versions_ in this example) since HBase version 0.1, so this is > going to be a source of confusion - I've seen it in practice. > A literate builder API, by way of the way we design its method names, could > let a user describe more or less in speaking language how they want version > retention to work, and internally the builder API could set the low level > schema attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema
[ https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23678: Description: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions after that interval elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. was: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. > Literate builder API for version management in schema > - > > Key: HBASE-23678 > URL: https://issues.apache.org/jira/browse/HBASE-23678 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and > KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance > regarding their usage. Almost all combinations of these four settings make > sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and > KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the > behavior with TTL easier to conceive when creating the schema. This could > take the form of a literate builder API for ColumnDescriptor or an extension > to an existing one. > Let me give you a motivating example: We may want to retain all versions for > a given TTL, and then only a specific number of versions after that interval > elapses. This can be achieved with VERSIONS=INT_MAX, > TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ > . This is not intuitive though because VERSIONS has been used to specify the > number of versions to retain (_num_versions_ in this example) since HBase > version 0.1, so this is going to be a source of confusion - I've seen it in > practice. > A literate builder API, by way of the way we design its method names, could > let a user describe more or less in speaking language how they want version > retention to work, and internally the builder API could set the low level > schema attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23678) Literate builder API for version management in schema
[ https://issues.apache.org/jira/browse/HBASE-23678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23678: Description: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions after that interval elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. was: Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance regarding their usage. Almost all combinations of these four settings make sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the behavior with TTL easier to conceive when creating the schema. This could take the form of a literate builder API for ColumnDescriptor or an extension to an existing one. Let me give you a motivating example: We may want to retain all versions for a given TTL, and then only a specific number of versions after that interval elapses. This can be achieved with VERSIONS=INT_MAX, TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSION=_num_versions_ . This is not intuitive though because VERSIONS has been used to specify the number of versions to retain (_num_versions_ in this example) since HBase version 0.1, so this is going to be a source of confusion - I've seen it in practice. A literate builder API, by way of the way we design its method names, could let a user describe more or less in speaking language how they want version retention to work, and internally the builder API could set the low level schema attributes. > Literate builder API for version management in schema > - > > Key: HBASE-23678 > URL: https://issues.apache.org/jira/browse/HBASE-23678 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > Lars designed the combination of VERSIONS, TTL, MIN_VERSIONS, and > KEEP_DELETED_CELLS with a maximum of flexibility. There is a lot of nuance > regarding their usage. Almost all combinations of these four settings make > sense for some use cases (exceptions are MIN_VERSIONS > 0 without TTL, and > KEEP_DELETED_CELLS=TTL without TTL). There should be a way to make the > behavior with TTL easier to conceive when creating the schema. This could > take the form of a literate builder API for ColumnDescriptor or an extension > to an existing one. > Let me give you a motivating example: We may want to retain all versions for > a given TTL, and then only a specific number of versions after that interval > elapses. This can be achieved with VERSIONS=INT_MAX, > TTL=_retention_interval_, KEEP_DELETED_CELLS=TTL, MIN_VERSIONS=_num_versions_ > . This is not intuitive though because VERSIONS has been used to specify the > number of versions to retain (_num_versions_ in this example) since HBase > version 0.1, so this is going to be a source of confusion - I've seen it in > practice. > A literate builder API, by way of the way we design its method names, could > let a user describe more or less in speaking language how they want version > retention to work, and internally the builder API could set the low level > schema attributes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled
Josh Elser created HBASE-23679: -- Summary: FileSystem instance leaks due to bulk loads with Kerberos enabled Key: HBASE-23679 URL: https://issues.apache.org/jira/browse/HBASE-23679 Project: HBase Issue Type: Bug Reporter: Josh Elser Assignee: Josh Elser Spent the better part of a week chasing an issue on HBase 2.x where the number of DistributedFileSystem instances on the heap of a RegionServer would grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an immense number of DFS instances cached (in FileSystem$Cache) for the same user, with the unique number of Tokens contained in that DFS's UGI member (one hbase delegation token, and two HDFS delegation tokens – we only do this for bulk loads). For the user's clusters, they eventually experienced 10x perf degradation as RegionServers spent all of their time in JVM GC (they were unlucky to not have RegionServers crash outright, as this would've, albeit temporarily, fixed the issue). The problem seems to be two-fold with changes by HBASE-15291 being largely the cause. This issue tried to close FileSystem instances which were being leaked – however, it did this by instrumenting the method {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this approach: 1. It relies on clients to call this method (client's hanging up will leak resources in RegionServers) 2. This method is only called on the RegionServer hosting the first Region of the table which was bulk-loaded into. For multiple RegionServers, they are left to leak resources. HBASE-21342 later tried to fix an issue where FS objects were now being closed prematurely via reference-counting (which appears to work fine), but does not address the other two issues above. Through all of this, I (re)learned the dirty history of UGI and how its caching doesn't work so great HADOOP-6670. I see trying to continue to leverage the FileSystem$CACHE as a potentially dangerous thing (we've been back here multiple times already). My opinion at this point is that we should cleanly create a new FileSystem instance during the call to {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally block in that same method. This both simplifies the lifecycle of a FileSystem instance in the bulk-load codepath but also helps us avoid future problems with UGI and FS caching. The one downside is that we pay the penalty to create a new FileSystem instance, but I'm of the opinion that we cross that bridge when we get there. Thanks for [~jdcryans] and [~busbey] for their help along the way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled
[ https://issues.apache.org/jira/browse/HBASE-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-23679: --- Description: Spent the better part of a week chasing an issue on HBase 2.x where the number of DistributedFileSystem instances on the heap of a RegionServer would grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an immense number of DFS instances cached (in FileSystem$Cache) for the same user, with the unique number of Tokens contained in that DFS's UGI member (one hbase delegation token, and two HDFS delegation tokens – we only do this for bulk loads). For the user's clusters, they eventually experienced 10x perf degradation as RegionServers spent all of their time in JVM GC (they were unlucky to not have RegionServers crash outright, as this would've, albeit temporarily, fixed the issue). The problem seems to be two-fold with changes by HBASE-15291 being largely the cause. This issue tried to close FileSystem instances which were being leaked – however, it did this by instrumenting the method {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this approach: # It relies on clients to call this method (client's hanging up will leak resources in RegionServers) # This method is only called on the RegionServer hosting the first Region of the table which was bulk-loaded into. For multiple RegionServers, they are left to leak resources. HBASE-21342 later tried to fix an issue where FS objects were now being closed prematurely via reference-counting (which appears to work fine), but does not address the other two issues above. Point #2 makes debugging this issue harder than normal because it doesn't manifest on a single node instance :) Through all of this, I (re)learned the dirty history of UGI and how its caching doesn't work so great HADOOP-6670. I see trying to continue to leverage the FileSystem$CACHE as a potentially dangerous thing (we've been back here multiple times already). My opinion at this point is that we should cleanly create a new FileSystem instance during the call to {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally block in that same method. This both simplifies the lifecycle of a FileSystem instance in the bulk-load codepath but also helps us avoid future problems with UGI and FS caching. The one downside is that we pay the penalty to create a new FileSystem instance, but I'm of the opinion that we cross that bridge when we get there. Thanks for [~jdcryans] and [~busbey] for their help along the way. was: Spent the better part of a week chasing an issue on HBase 2.x where the number of DistributedFileSystem instances on the heap of a RegionServer would grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we had an immense number of DFS instances cached (in FileSystem$Cache) for the same user, with the unique number of Tokens contained in that DFS's UGI member (one hbase delegation token, and two HDFS delegation tokens – we only do this for bulk loads). For the user's clusters, they eventually experienced 10x perf degradation as RegionServers spent all of their time in JVM GC (they were unlucky to not have RegionServers crash outright, as this would've, albeit temporarily, fixed the issue). The problem seems to be two-fold with changes by HBASE-15291 being largely the cause. This issue tried to close FileSystem instances which were being leaked – however, it did this by instrumenting the method {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this approach: 1. It relies on clients to call this method (client's hanging up will leak resources in RegionServers) 2. This method is only called on the RegionServer hosting the first Region of the table which was bulk-loaded into. For multiple RegionServers, they are left to leak resources. HBASE-21342 later tried to fix an issue where FS objects were now being closed prematurely via reference-counting (which appears to work fine), but does not address the other two issues above. Through all of this, I (re)learned the dirty history of UGI and how its caching doesn't work so great HADOOP-6670. I see trying to continue to leverage the FileSystem$CACHE as a potentially dangerous thing (we've been back here multiple times already). My opinion at this point is that we should cleanly create a new FileSystem instance during the call to {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally block in that same method. This both simplifies the lifecycle of a FileSystem instance in the bulk-load codepath but also helps us avoid future problems with UGI and FS caching. The one downside is that we pay the penalty to create a new FileSystem instance, but I'm of the opinion that we cross that bridge when we get there. Thanks for [~jdcryans] and [~busbey] for their
[GitHub] [hbase] ndimiduk merged pull request #1010: HBASE-23664 Upgrade JUnit to 4.13
ndimiduk merged pull request #1010: HBASE-23664 Upgrade JUnit to 4.13 URL: https://github.com/apache/hbase/pull/1010 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23628) Remove Apache Commons Digest Base64
[ https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23628: Component/s: dependencies > Remove Apache Commons Digest Base64 > --- > > Key: HBASE-23628 > URL: https://issues.apache.org/jira/browse/HBASE-23628 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 3.0.0 > > > Use the native JDK Base64 implementation instead. Most places are using the > JDK version, but a couple of spots were missed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] joshelser opened a new pull request #1019: HBASE-23679 Use new FileSystem objects during bulk loads
joshelser opened a new pull request #1019: HBASE-23679 Use new FileSystem objects during bulk loads URL: https://github.com/apache/hbase/pull/1019 Undoes lots of fanciness about FileSystem caching because of an explicit bug that was present (FileSystem only closed on one out of N regionservers, and not every time), but also because the FS caching itself is dodgy and prone to error. Each BulkLoad request will now get its own FileSystem instance that it is responsible for closing. The change to HRegion is because ${hbase.rootdir}/data is `chmod 700` which means that a normal user cannot get the size of those files (you'll see lots of AccessDenied exceptions in the RS log). Using HRegionFilesystem instead keeps this from being an issue (reading the filesize as HBase instead of the user). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-23664) Upgrade JUnit to 4.13
[ https://issues.apache.org/jira/browse/HBASE-23664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-23664. -- Resolution: Fixed > Upgrade JUnit to 4.13 > - > > Key: HBASE-23664 > URL: https://issues.apache.org/jira/browse/HBASE-23664 > Project: HBase > Issue Type: Task > Components: integration tests, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > New JUnit released a week ago. Let's give it a spin. > https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.13.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23676) Address feedback on HBASE-23055 Alter hbase:meta.
[ https://issues.apache.org/jira/browse/HBASE-23676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013220#comment-17013220 ] Michael Stack commented on HBASE-23676: --- {code} Duo Zhang 5:33 AM (8 hours ago) to State, apache/hbase, Michael @Apache9 commented on this pull request. In hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java: > @@ -666,42 +667,55 @@ public void run(PRESP resp) { new DisableTableProcedureBiConsumer(tableName)); } + /** + * Utility for completing passed TableState {@link CompletableFuture} future + * using passed parameters. + */ + private static CompletableFuture completeCheckTableState( What is the return value used for? In hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java: > @@ -58,8 +62,19 @@ private final ZNodePaths znodePaths; + /** + * A znode maintained by MirroringTableStateManager. + * MirroringTableStateManager is deprecated to be removed in hbase3. It can also be disabled. + * Make sure it is enabled if you want to alter hbase:meta table in hbase2. In hbase3, + * TBD how metatable state will be hosted; likely on active hbase master. I think the state will either be on zk or on hdfs? HMaster itself is state less... In general, in the current architecture, I think it the state should be placed on zk, of course you could cache it in master and let client go to master to ask for the state. In hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java: > @@ -58,8 +62,19 @@ private final ZNodePaths znodePaths; + /** + * A znode maintained by MirroringTableStateManager. + * MirroringTableStateManager is deprecated to be removed in hbase3. It can also be disabled. + * Make sure it is enabled if you want to alter hbase:meta table in hbase2. In hbase3, + * TBD how metatable state will be hosted; likely on active hbase master. So I do not think we should name this as mirrored, it is the original data. The state in master's memory, is a cache, actually. In hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java: > @@ -229,6 +244,43 @@ private void > getMetaRegionLocation(CompletableFuture future, }); } + @Override + public CompletableFuture getMetaTableState() { +return getAndConvert(this.znodeMirroredMetaTableState, ZKAsyncRegistry::getTableState). + thenApply(state -> { +return state == null || state.equals(ENABLED_META_TABLE_STATE.getState())? + ENABLED_META_TABLE_STATE: new TableState(TableName.META_TABLE_NAME, state); + }).exceptionally(e -> { Currently in HBase, usually we will create a new CompletableFuture and use FutureUtils.addListener to complete it. The code in exceptionally are a bit tricky, where we throw a CompletionException... In hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java: >private static final Logger LOG = > LoggerFactory.getLogger(TableStateManager.class); + + /** + * All table state is kept in hbase:meta except that of hbase:meta itself. In general I do not think this is a good design. We should persist it somewhere, and just cache it in master's memory. And we do not need to disable a table when altering any more? And the solution here just assume that we only have one meta region? So the altering operation can be atomic? What if we have multiple meta regions and we crash in the middle? I think this patch also aims to implement splittable meta in the future? No? In hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RejectReplicationRequestStateChecker.java: > @@ -1,4 +1,6 @@ -/** +/* + * Copyright The Apache Software Foundation What's this? In hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseMetaEdit.java: > +Admin admin = UTIL.getAdmin(); +admin.tableExists(TableName.META_TABLE_NAME); +admin.disableTable(TableName.META_TABLE_NAME); +assertTrue(admin.isTableDisabled(TableName.META_TABLE_NAME)); +TableDescriptor descriptor = admin.getDescriptor(TableName.META_TABLE_NAME); +ColumnFamilyDescriptor cfd = descriptor.getColumnFamily(HConstants.CATALOG_FAMILY); +byte [] extraColumnFamilyName = Bytes.toBytes("xtra"); +ColumnFamilyDescriptor newCfd = + ColumnFamilyDescriptorBuilder.newBuilder(extraColumnFamilyName).build(); +int oldVersions = cfd.getMaxVersions(); +// Add '1' to current versions count. +cfd = ColumnFamilyDescriptorBuilder.newBuilder(cfd).setMaxVersions(oldVersions + 1). +setConfiguration(ColumnFamilyDescriptorBuilder.DATA_BLOCK_ENCODING, +DataBlockEncoding.ROW_INDEX_V1.toString()).build(); +admin.modifyColumnFamily(TableName.META_TABLE_NAME, cfd); +admin.addColumnFamily(TableName.META_TABLE_NAME, newCfd); Now we allow users to add/remove fam
[jira] [Reopened] (HBASE-23628) Remove Apache Commons Digest Base64
[ https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-23628: - reopening for backport(s) > Remove Apache Commons Digest Base64 > --- > > Key: HBASE-23628 > URL: https://issues.apache.org/jira/browse/HBASE-23628 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 3.0.0 > > > Use the native JDK Base64 implementation instead. Most places are using the > JDK version, but a couple of spots were missed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23628) Remove Apache Commons Digest Base64
[ https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23628. - Fix Version/s: 2.3.0 Resolution: Fixed > Remove Apache Commons Digest Base64 > --- > > Key: HBASE-23628 > URL: https://issues.apache.org/jira/browse/HBASE-23628 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Use the native JDK Base64 implementation instead. Most places are using the > JDK version, but a couple of spots were missed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23628) Replace Apache Commons Digest Base64 with JDK8 Base64
[ https://issues.apache.org/jira/browse/HBASE-23628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23628: Summary: Replace Apache Commons Digest Base64 with JDK8 Base64 (was: Remove Apache Commons Digest Base64) > Replace Apache Commons Digest Base64 with JDK8 Base64 > - > > Key: HBASE-23628 > URL: https://issues.apache.org/jira/browse/HBASE-23628 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Use the native JDK Base64 implementation instead. Most places are using the > JDK version, but a couple of spots were missed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…
Apache-HBase commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… URL: https://github.com/apache/hbase/pull/1012#issuecomment-573226238 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 37s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -0 :warning: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 31s | master passed | | +1 :green_heart: | compile | 1m 0s | master passed | | +1 :green_heart: | checkstyle | 1m 44s | master passed | | +1 :green_heart: | shadedjars | 5m 25s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 40s | master passed | | +0 :ok: | spotbugs | 5m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 5m 0s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 52s | the patch passed | | +1 :green_heart: | compile | 1m 5s | the patch passed | | +1 :green_heart: | javac | 1m 5s | the patch passed | | +1 :green_heart: | checkstyle | 1m 39s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 18m 30s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | +1 :green_heart: | javadoc | 0m 38s | the patch passed | | +1 :green_heart: | findbugs | 4m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 156m 1s | hbase-server in the patch passed. | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 223m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/1012 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 888eea02fdcf 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1012/out/precommit/personality/provided.sh | | git revision | master / 8ca614857d | | Default Java | 1.8.0_181 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/testReport/ | | Max. process+thread count | 5889 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1012/2/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013235#comment-17013235 ] Michael Stack commented on HBASE-23326: --- [~zhangduo] I see that the regionprocedurestore WAL files are being moved to the /hbase/archive but I also see build up at /hbase/MasterProcs/archive of old procedure hfiles. I have 1500 files in it currently (running about 24 hours under load). Poking in PR I don't see obvious handling of this Region archive dir. What am I missing? Thanks. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters…
saintstack commented on issue #1012: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… URL: https://github.com/apache/hbase/pull/1012#issuecomment-573230752 Ok. So changing blooms seems fine. Let me retry with the encoder. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013250#comment-17013250 ] Hudson commented on HBASE-18095: Results for branch HBASE-18095/client-locate-meta-no-zookeeper [build #35 on builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//General_Nightly_Build_Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//JDK8_Nightly_Build_Report_(Hadoop2)/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18095%252Fclient-locate-meta-no-zookeeper/35//JDK8_Nightly_Build_Report_(Hadoop3)/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23349) Low refCount preventing archival of compacted away files
[ https://issues.apache.org/jira/browse/HBASE-23349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013255#comment-17013255 ] Andrew Kyle Purtell commented on HBASE-23349: - {quote} I think we should step back and remember why we have the ref counting in the first place. This came from a discussion started in HBASE-13082 and HBASE-10060, namely too much synchronization. If any changes we make now needs new synchronization in the scanner.next(...) path we're back to where we started and in that case we should remove the ref counting and bring back the old notification and scanner switching we had before. {quote} I made a similar comment on an internal discussion yesterday. If we have to walk back the StoreScanner "lock coarsening" work, then let's not be afraid to do it. There is a nuanced decision we would have to make, but let's not be concerned about sunk costs. > Low refCount preventing archival of compacted away files > > > Key: HBASE-23349 > URL: https://issues.apache.org/jira/browse/HBASE-23349 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.3.0, 1.6.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > We have observed that refCount on compacted away store files as low as 1 is > prevent archival. > {code:java} > regionserver.HStore - Can't archive compacted file > hdfs://{{root-dir}}/hbase/data/default/t1/12a9e1112e0371955b3db8d3ebb2d298/cf1/73b72f5ddfce4a34a9e01afe7b83c1f9 > because of either isCompactedAway=true or file has reference, > isReferencedInReads=true, refCount=1, skipping for now. > {code} > We should come up with core code (run as part of discharger thread) > gracefully resolve reader lock issue by resetting ongoing scanners to start > pointing to new store files instead of compacted away store files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1020: HBASE-23653 Expose content of meta table in web ui
ndimiduk opened a new pull request #1020: HBASE-23653 Expose content of meta table in web ui URL: https://github.com/apache/hbase/pull/1020 Adds a display of the content of 'hbase:meta' to the Master's table.jsp, when that table is selected. Supports basic pagination, filtering, &c. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23653) Expose content of meta table in web ui
[ https://issues.apache.org/jira/browse/HBASE-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-23653: - Attachment: 23653.1.png 23653.0.png > Expose content of meta table in web ui > -- > > Key: HBASE-23653 > URL: https://issues.apache.org/jira/browse/HBASE-23653 > Project: HBase > Issue Type: New Feature > Components: master, Operability, UI >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 23653.0.png, 23653.1.png > > > While testing improvements to hbck2, breaking and fixing a cluster's region > assignments, we've found that quite a few questions can only be answered by > dropping into a shell and scanning meta. This is in contrast to the nice web > view we have on procedures and their locks. It would be nice to have a > similar view into the content of meta. I've put together and extension to the > existing {{table.jsp}} that does this. It's had a couple rounds of operator > feedback. Will post up some screenshots along with the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23679) FileSystem instance leaks due to bulk loads with Kerberos enabled
[ https://issues.apache.org/jira/browse/HBASE-23679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013260#comment-17013260 ] Josh Elser commented on HBASE-23679: Dropping some more details in here, should anyone else ever find themselves back here. * Funny that this user's RegionServers never (rarely?) actually OOME'ed. They were left in this crap state because the JVM just kept hanging on. GC verbose logging showed us that heap was just sitting completely full. We just sat with 30-60s GC pauses, getting next to nothing done. * A heapdump showed (at worst) 60k+ instances of DistributedFileSystem sitting on the heap, nearly all of them referenced by {{FileSystem$CACHE}}. It takes some digging to realize this. * Digging into each DFS -> DFSClient -> UGI -> Tokens, you can see the delegation tokens which let us confirm that Bulk Load was the culprit creating/leaking these. * For the user, it was an extremely slow growth. One RegionServer would get into this state every ~20 days across about that many RegionServers. So, in reality, we were probably looking at even longer than 20days for a single RegionServer to leak enough of these to cause issues. * UGI being a part of FileSystem caching is crap, and we should avoid it for anything other than "just the HBase user". It's extremely obtuse and error prone, likely to just give us more heartache (lots of issues around this in the past). > FileSystem instance leaks due to bulk loads with Kerberos enabled > - > > Key: HBASE-23679 > URL: https://issues.apache.org/jira/browse/HBASE-23679 > Project: HBase > Issue Type: Bug >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > > Spent the better part of a week chasing an issue on HBase 2.x where the > number of DistributedFileSystem instances on the heap of a RegionServer would > grow unbounded. Looking at multiple heap-dumps, it was obvious to see that we > had an immense number of DFS instances cached (in FileSystem$Cache) for the > same user, with the unique number of Tokens contained in that DFS's UGI > member (one hbase delegation token, and two HDFS delegation tokens – we only > do this for bulk loads). For the user's clusters, they eventually experienced > 10x perf degradation as RegionServers spent all of their time in JVM GC (they > were unlucky to not have RegionServers crash outright, as this would've, > albeit temporarily, fixed the issue). > The problem seems to be two-fold with changes by HBASE-15291 being largely > the cause. This issue tried to close FileSystem instances which were being > leaked – however, it did this by instrumenting the method > {{SecureBulkLoadManager.cleanupBulkLoad(..)}}. Two big issues with this > approach: > # It relies on clients to call this method (client's hanging up will leak > resources in RegionServers) > # This method is only called on the RegionServer hosting the first Region of > the table which was bulk-loaded into. For multiple RegionServers, they are > left to leak resources. > HBASE-21342 later tried to fix an issue where FS objects were now being > closed prematurely via reference-counting (which appears to work fine), but > does not address the other two issues above. Point #2 makes debugging this > issue harder than normal because it doesn't manifest on a single node > instance :) > Through all of this, I (re)learned the dirty history of UGI and how its > caching doesn't work so great HADOOP-6670. I see trying to continue to > leverage the FileSystem$CACHE as a potentially dangerous thing (we've been > back here multiple times already). My opinion at this point is that we should > cleanly create a new FileSystem instance during the call to > {{SecureBulkLoadManager#secureBulkLoadHFiles(..)}} and close it in a finally > block in that same method. This both simplifies the lifecycle of a FileSystem > instance in the bulk-load codepath but also helps us avoid future problems > with UGI and FS caching. The one downside is that we pay the penalty to > create a new FileSystem instance, but I'm of the opinion that we cross that > bridge when we get there. > Thanks for [~jdcryans] and [~busbey] for their help along the way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23653) Expose content of meta table in web ui
[ https://issues.apache.org/jira/browse/HBASE-23653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013262#comment-17013262 ] Nick Dimiduk commented on HBASE-23653: -- Here's a couple screen shots of the new meta browser. > Expose content of meta table in web ui > -- > > Key: HBASE-23653 > URL: https://issues.apache.org/jira/browse/HBASE-23653 > Project: HBase > Issue Type: New Feature > Components: master, Operability, UI >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Attachments: 23653.0.png, 23653.1.png > > > While testing improvements to hbck2, breaking and fixing a cluster's region > assignments, we've found that quite a few questions can only be answered by > dropping into a shell and scanning meta. This is in contrast to the nice web > view we have on procedures and their locks. It would be nice to have a > similar view into the content of meta. I've put together and extension to the > existing {{table.jsp}} that does this. It's had a couple rounds of operator > feedback. Will post up some screenshots along with the PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing
ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing URL: https://github.com/apache/hbase/pull/917#discussion_r365466674 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java ## @@ -86,74 +84,133 @@ void fix() throws IOException { * If hole, it papers it over by adding a region in the filesystem and to hbase:meta. * Does not assign. */ - void fixHoles(CatalogJanitor.Report report) throws IOException { -List> holes = report.getHoles(); + void fixHoles(CatalogJanitor.Report report) { +final List> holes = report.getHoles(); if (holes.isEmpty()) { - LOG.debug("No holes."); + LOG.info("CatalogJanitor Report contains no holes to fix. Skipping."); Review comment: Good sit, this log line is emitted by `MetaFixer`, speaking of its input. It'll be run just when an operator has invoked hbck2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing
ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing URL: https://github.com/apache/hbase/pull/917#discussion_r365466933 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java ## @@ -86,74 +84,133 @@ void fix() throws IOException { * If hole, it papers it over by adding a region in the filesystem and to hbase:meta. * Does not assign. */ - void fixHoles(CatalogJanitor.Report report) throws IOException { -List> holes = report.getHoles(); + void fixHoles(CatalogJanitor.Report report) { +final List> holes = report.getHoles(); if (holes.isEmpty()) { - LOG.debug("No holes."); + LOG.info("CatalogJanitor Report contains no holes to fix. Skipping."); return; } -for (Pair p: holes) { - RegionInfo ri = getHoleCover(p); - if (ri == null) { -continue; - } - Configuration configuration = this.masterServices.getConfiguration(); - HRegion.createRegionDir(configuration, ri, FSUtils.getRootDir(configuration)); - // If an error here, then we'll have a region in the filesystem but not - // in hbase:meta (if the below fails). Should be able to rerun the fix. - // Add to hbase:meta and then update in-memory state so it knows of new - // Region; addRegionToMeta adds region and adds a state column set to CLOSED. - MetaTableAccessor.addRegionToMeta(this.masterServices.getConnection(), ri); - this.masterServices.getAssignmentManager().getRegionStates(). - updateRegionState(ri, RegionState.State.CLOSED); - LOG.info("Fixed hole by adding {} in CLOSED state; region NOT assigned (assign to ONLINE).", - ri); -} + +LOG.info("Identified {} region holes to fix. Detailed fixup progress logged at DEBUG.", Review comment: I want the system to log at least acknowledgement that the operator requested a run. That's the above log line -- an empty CatalogJanitor Report means no work to do. This line is setting the stage for how much work it thinks needs done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing
ndimiduk commented on a change in pull request #917: HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing URL: https://github.com/apache/hbase/pull/917#discussion_r365467011 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java ## @@ -86,74 +84,133 @@ void fix() throws IOException { * If hole, it papers it over by adding a region in the filesystem and to hbase:meta. * Does not assign. */ - void fixHoles(CatalogJanitor.Report report) throws IOException { -List> holes = report.getHoles(); + void fixHoles(CatalogJanitor.Report report) { +final List> holes = report.getHoles(); if (holes.isEmpty()) { - LOG.debug("No holes."); + LOG.info("CatalogJanitor Report contains no holes to fix. Skipping."); return; } -for (Pair p: holes) { - RegionInfo ri = getHoleCover(p); - if (ri == null) { -continue; - } - Configuration configuration = this.masterServices.getConfiguration(); - HRegion.createRegionDir(configuration, ri, FSUtils.getRootDir(configuration)); - // If an error here, then we'll have a region in the filesystem but not - // in hbase:meta (if the below fails). Should be able to rerun the fix. - // Add to hbase:meta and then update in-memory state so it knows of new - // Region; addRegionToMeta adds region and adds a state column set to CLOSED. - MetaTableAccessor.addRegionToMeta(this.masterServices.getConnection(), ri); - this.masterServices.getAssignmentManager().getRegionStates(). - updateRegionState(ri, RegionState.State.CLOSED); - LOG.info("Fixed hole by adding {} in CLOSED state; region NOT assigned (assign to ONLINE).", - ri); -} + +LOG.info("Identified {} region holes to fix. Detailed fixup progress logged at DEBUG.", + holes.size()); + +final List newRegionInfos = createRegionInfosForHoles(holes); +final List newMetaEntries = createMetaEntries(masterServices, newRegionInfos); +final TransitRegionStateProcedure[] assignProcedures = masterServices + .getAssignmentManager() + .createRoundRobinAssignProcedures(newMetaEntries); + + masterServices.getMasterProcedureExecutor().submitProcedures(assignProcedures); +LOG.info( Review comment: The procedure engine logs a ton, but how will the operator know which procIds are the the ones that've been requested by this fixup? I'd think it's good to keep this message, it's an effective summary of the work done. Make it DEBUG though? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013270#comment-17013270 ] Michael Stack commented on HBASE-23326: --- And this seem to be hanging around... %2C16000%2C1578524235076.1578524242389$masterproc$ though straight WALs are cleared. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #1021: Backport HBASE-23653 Expose content of meta table in web ui
ndimiduk opened a new pull request #1021: Backport HBASE-23653 Expose content of meta table in web ui URL: https://github.com/apache/hbase/pull/1021 Adds a display of the content of 'hbase:meta' to the Master's table.jsp, when that table is selected. Supports basic pagination, filtering, &c. Backport of #1020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services