[
https://issues.apache.org/jira/browse/HBASE-24246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090978#comment-17090978
]
Michael Stack commented on HBASE-24246:
---------------------------------------
Address all but the compaction issue. Thats for another JIRA.
> Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around
> merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-24246
> URL: https://issues.apache.org/jira/browse/HBASE-24246
> Project: HBase
> Issue Type: Bug
> Reporter: Michael Stack
> Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> Bit of a grab bag of small items.
> Improve log messages when bulk merge runs (particularly out of 'hbck2
> fixMeta').
> Change config so we up the merge maximum since current low max is hit easily.
> When its hit, its merge of merge of merges w/ confusing output while
> happening. LOG when we hit max!
> {code}
> private static final String MAX_MERGE_COUNT_KEY =
> "hbase.master.metafixer.max.merge.count";
> private static final int MAX_MERGE_COUNT_DEFAULT = 10;
> {code}
> Fix the 'HBCK Report' page so it only logs encoded region names because if
> region name is large, the UI becomes unreadable.
> Fixup some logging messages. As is, can confuse operator.
> E.g. explain why having a merge qualifier blocks a region being merged up
> {code}
> org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging
> [e1ded5983cb00c1a1a45e17414c5914c, 2634c71a10cf4a216b7c1608b7d24e6a,
> e7d40bb7ab50cdeb20118043aedf2cdc, a9ea228464bd8a40b8908cd263e2f6be,
> 544d32880131443eebb2a01691994b0d, a82d462a6edca634e4f8ecff4e8bc55c,
> 03e30d277a1ab977f93c103cb917c7d7, d0d5fb364838b00942448b89b3d802ff,
> 00ba78a9be4b27cf076622f49c45a43e, 338efcfc48928815423d7a6601b9e477], because
> parent [e1ded5983cb00c1a1a45e17414c5914c] has a merge qualifier
> {code}
> Add note to orphans page that you shouldn't try to address orphans until all
> meta issues are fixed (overlaps, holes).
> Add example on how to fix orphans with bulkload to the orphans section in
> hbck report. See HBASE-23928.
> Why the angle brackets in here:
> {code}
> Why angle brackets here? Usually for lists.
> 2020-04-23 15:21:13,471 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=hbase//10.103.230.91 merge regions [e1ded5983cb00c1a1a45e17414c5914c],
> [2634c71a10cf4a216b7c1608b7d24e6a], [e7d40bb7ab50cdeb20118043aedf2cdc],
> [a9ea228464bd8a40b8908cd263e2f6be], [544d32880131443eebb2a01691994b0d],
> [a82d462a6edca634e4f8ecff4e8bc55c], [03e30d277a1ab977f93c103cb917c7d7],
> [d0d5fb364838b00942448b89b3d802ff], [50cbee16764bb09a9887a07d3a01304f],
> [338efcfc48928815423d7a6601b9e477]
> {code}
> These logs make no sense... need region name at least
> {code}
> What is this? Missing region
> Whats it doing?
> 2020-04-23 15:36:19,896 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=562814154
> 2020-04-23 15:36:22,953 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=564904980
> 2020-04-23 15:36:26,011 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=566948397
> 2020-04-23 15:36:29,071 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=568943946
> 2020-04-23 15:36:32,153 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=570961155
> 2020-04-23 15:36:35,219 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=573055384
> 2020-04-23 15:36:38,274 WARN
> org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
> totalCompactingKVs=500851692 less than currentCompactedKVs=575099844
> {code}
> This 'no state' is confusing... should be unknown region.
> {code}
> Why this message
> 2020-04-22 23:34:14,617 ERROR
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error
> trying to merge [5aee12dd3e126b1fc4f5884be9712a5e,
> 1b4540d46c8ae2221759c72e328e3b28, 16f90525635f972bc984a498a17ceca4,
> f22469447941caf3d885b860d2f9ac8b, 971e35bff0c4edfd5416db246f499ac9,
> 441010c4577ee79c07a857e507567dc2, 94eaaee33dd2d22eebf40dec418e1f1c,
> d9a4fce4cc742a4d79cffb67a1b77510, b136369ec08d84d3761dcdde511c32e1,
> a6f6f29ce45d368fe49651723cacce10] in bobby_analytics (in
> state=MERGE_TABLE_REGIONS_PREPARE)
> org.apache.hadoop.hbase.UnknownRegionException: No state for
> [5aee12dd3e126b1fc4f5884be9712a5e]
> at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:487)
> at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)
> at
> org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)
> {code}
> Fix this log so can just copy paste it to shell... fix quotes:
> org.apache.hadoop.hbase.exceptions.MergeRegionException: Unable to merge
> non-adjacent or non-overlapping regions 9090c9927244d39d08dcf24335d2aa48,
> 04dc9c18f74937d2359e4ac2aaee9937 when force=false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
