On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad <mohammad.ars...@huawei.com> wrote:
> Hello HBase Folks > > Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to > be fixed by manually by executing HBCK2 commands. > HBCK2 requires high HBase skills. It is bit difficult for maintenance > personals to figure out which command, when and in which order to be > executed. > > True. Is there any effort going on in community to automate fixing these > inconsistencies? I also would like to contribute there. > > I was thinking, maybe we can expose CJ and HBCK chore reported > inconsistencies through a new master API and then provide option to fix > these inconsistencies. Basically adding two new commands in HBCK2 > -listInconsistencies list CJ and HBCK chore reported inconsistencies > -fixInconsistencies fix CJ and HBCK chore reported inconsistencies (Not > sure if possible to fix all inconsistencies, need to analyze all > inconsistencies case by case, but some are very straight forward for > example holes and overlap) > > So, a 'god' command that will fix any issue found? That is tough. You've seen the philosophy section on hbck2, of how it makes no claims to being so capable [1]? We are trying to get to a place where hbck2 is increasingly less necessary. The general idea is that inconsistencies are caused by bugs or oversight. As time goes by, we've been plugging the holes. Upgrading hbase gains you the fixes making the need for hbck2 less. But as you state above, when there is an issue, it can be hard for the operator to figure how to make fixes. We've been trying to improve this state with documentation in the UI up on the 'HBCK Report' page and elsewhere but there is room for improvement. We've also been trying to aggregate on the hbck2 side so that commands become increasingly 'macro', fixing a whole category of problem types rather than an affliction at a time. This should make the tool easier to use. The 'fixMeta' command is a good example here as it fixes any holes or overlaps found in hbase:meta (This is probably ripe for conversion into an auto-repair run on occasion by the Master). Another way in which we've been trying to make improvement is by obsoleting commands in hbck2 as we fix the root cause that required the hbck2 command option to be needed in the first place. CJ and the HBCK Chore can report on inconsistencies found. It is another thing altogether having them go ahead and repair any issues found mostly because we are not yet confident the repair won't cause more damage than it fixes. > Any thoughts/inputs highly appreciated. > > Are you regularly experiencing consistency issues? If so, what version of hbase are you running? Thanks, S 1. https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy > Regards > -Arshad > > >
