I recently added a shell command "rit" that displays the list of current RIT. Would that have worked? It does require that the master is responsive to a GetClusterStatus request.
> On Dec 6, 2018, at 7:45 AM, Sean Busbey <[email protected]> wrote: > > This week I've run into two cases where I needed the set of regions in > transition so I could recover them and I ran into what I think is a > gap in our operator tooling. I'm hoping folks will have some ideas > I've missed. > > Depending on how this thread goes, I'll make some follow-on on the > dev@hbase list for implementing changes and documentation. > > Case 1: HBase 1.2-ish RIT following RS crash > > Cluster had a handful of region servers fail and for whatever reason a > few regions were stuck in transition. The operator I was helping > already is used to dealing with the occasional manual recovery. Their > normal process looks like this: > > 1) Got to Master UI website > 2) Scroll down to Regions in Transition list > 3) Find a RIT in FAILED_CLOSE / FAILED_OPEN / PENDING_OPEN > 4) confirm on RS logs that the RS associated in the above is now in > good health and doesn't expect to do anything with said region > 5) run "assign" in the hbase shell for the region > > Unfortunately, the cluster's HDFS was under duress and so listing > snapshot information was super slow. This caused the Master UI website > to hang prior to displaying the RIT list. > > We ended up looking at the master log file. > > Case 2: HBase 2.1-ish RIT following cluster wide crash > > AFAICT cluster had experienced a failure of all RS and masters. Upon > coming back up Master was left with ~10% of ~10K regions in a state of > PENDING_OPEN or OPENING all with a RS that had no idea it was involved > with those regions. I'm pretty sure this is a bug; I'm still triaging > it and I don't think it's relevant to the current question. > > Once I confirmed the given RS was not currently doing anything for any > of those regions I figured I'd use HBCK2 to run an assigns to get > things fixed. However, since there were like 900 RITs, the Master UI > was unusable for getting a complete list. Also with that many all in > the same state I want to be able to automate running against each of > them. > > I ended up greping the master log file and pulling out the WARN > messages about RIT to tease out the list of regions, then passed those > to hbck2. > > ---- > > Am I missing some obvious place where I can use a CLI tool to get a > list of RIT? I don't see anything in the ref guide. I looked through > the help of HBCK 1 and the shell and couldn't find anything. > > I think I can use Admin.getClusterStatus() and getClusterMetrics() to > get this info from the Java API. That means there's some way to get it > in the hbase shell, but it'll probably be ugly. If there's not already > an easier way I'll want to wrap that so it's a simple command.
