bq. And having a list of "dead" servers is not a healthy thing to have. I don't think the existence of "dead" servers means the service is unhealthy, especially in a distributed system. Besides hbase, HDFS also shows Live and Dead nodes in namenode UI, and people won't regard HDFS as unhealthy if there're dead nodes.
In HBase, if some RS aborts due to unexpected issue like long GC, normally we will restart it and once it's restarted and report to master, it will be removed from the dead server list. So when we observed dead server in Master UI, the first thing is to check the root cause and restart it if it won't cause further issue. However, sometimes we may find the server aborted due to some hardware failure and we must offline the server for repairing. Or we need to move some nodes to join other clusters so we stop the RS process on purpose. I guess this is the case you're dealing with @jeff? If so, I think it's a reasonable requirement that we supply a command in hbase to clear the dead nodes when operator assure they no longer serves. Best Regards, Yu On 27 May 2017 at 04:49, Enis Söztutar <enis....@gmail.com> wrote: > In general if there are no regions in transition, the WAL recovery has > already finished. You can watch the master's log4j log for those entries, > but the lack of regions in transition is the easiest way to identify. > > Enis > > On Fri, May 26, 2017 at 12:14 PM, jeff saremi <jeffsar...@hotmail.com> > wrote: > > > thanks Enis > > > > I apologize for earlier > > > > This looks very close to our issue > > When you say: "there is no "WAL" recovery is happening", how could i make > > sure of that? Thanks > > > > Jeff > > > > > > ________________________________ > > From: Enis Söztutar <enis....@gmail.com> > > Sent: Friday, May 26, 2017 11:47:11 AM > > To: d...@hbase.apache.org > > Cc: hbase-user > > Subject: Re: What is Dead Region Servers and how to clear them up? > > > > Jeff, please be respectful to be people who are trying to help you. This > is > > not acceptable behavior and will result in consequences next time. > > > > On the specific issue that you are seeing, it is highly likely that you > are > > seeing this: https://issues.apache.org/jira/browse/HBASE-14223. Having > > those servers in the dead servers list will not hurt operations, or > > runtimes or anything else. Possibly for those servers, there is not new > > instance of the regionserver running in the same host and ports. > > > > If you want to manually clean out these, you can follow these steps: > > - Manually move these directries from the file system: > > <hbase_hdfs>/WALs/dead-server-splitting > > - ONLY do this if you are sure that there is no "WAL" recovery is > > happening, and there is only WAL files with names containing ".meta." > > - Restart HBase master. > > > > Upon restart, you can see that these do not show up anymore. For more > > technical details, please refer to the jira link. > > > > Enis > > > > On Fri, May 26, 2017 at 11:03 AM, jeff saremi <jeffsar...@hotmail.com> > > wrote: > > > > > Thank you for the GFY answer > > > > > > And i guess to figure out how to fix these I can always go through the > > > HBase source code. > > > > > > > > > ________________________________ > > > From: Dima Spivak <dimaspi...@apache.org> > > > Sent: Friday, May 26, 2017 9:58:00 AM > > > To: hbase-user > > > Subject: Re: What is Dead Region Servers and how to clear them up? > > > > > > Sending this back to the user mailing list. > > > > > > RegionServers can die for many reasons. Looking at your RegionServer > log > > > files should give hints as to why it's happening. > > > > > > > > > -Dima > > > > > > On Fri, May 26, 2017 at 9:48 AM, jeff saremi <jeffsar...@hotmail.com> > > > wrote: > > > > > > > I had posted this to the user mailing list and I have not got any > > direct > > > > answer to my question. > > > > > > > > Where do dead RS's come from and how can they be cleaned up? Someone > in > > > > the midst of developers should know this. > > > > > > > > thanks > > > > > > > > Jeff > > > > > > > > ________________________________ > > > > From: jeff saremi <jeffsar...@hotmail.com> > > > > Sent: Thursday, May 25, 2017 10:23:17 AM > > > > To: user@hbase.apache.org > > > > Subject: Re: What is Dead Region Servers and how to clear them up? > > > > > > > > I'm still looking to get hints on how to remove the dead regions. > > thanks > > > > > > > > ________________________________ > > > > From: jeff saremi <jeffsar...@hotmail.com> > > > > Sent: Wednesday, May 24, 2017 12:27:06 PM > > > > To: user@hbase.apache.org > > > > Subject: Re: What is Dead Region Servers and how to clear them up? > > > > > > > > i'm trying to eliminate the dead region servers. > > > > > > > > ________________________________ > > > > From: Ted Yu <yuzhih...@gmail.com> > > > > Sent: Wednesday, May 24, 2017 12:17:40 PM > > > > To: user@hbase.apache.org > > > > Subject: Re: What is Dead Region Servers and how to clear them up? > > > > > > > > bq. running hbck (many times > > > > > > > > Can you describe the specific inconsistencies you were trying to > > resolve > > > ? > > > > Depending on the inconsistencies, advice can be given on the best > known > > > > hbck command arguments to use. > > > > > > > > Feel free to pastebin master log if needed. > > > > > > > > On Wed, May 24, 2017 at 12:10 PM, jeff saremi < > jeffsar...@hotmail.com> > > > > wrote: > > > > > > > > > these are the things I have done so far: > > > > > > > > > > > > > > > - restarting master (few times) > > > > > > > > > > - running hbck (many times; this tool does not seem to be doing > > > anything > > > > > at all) > > > > > > > > > > - checking the list of region servers in ZK (none of the dead ones > > are > > > > > listed here) > > > > > > > > > > - checking the WALs under <hbase_hdfs>/WALs. Out of 11 dead ones > > only 3 > > > > > are listed here with "-splitting" at the end of their names and > they > > > > > contain one single file like: 1493846660401..meta. > 1493922323600.meta > > > > > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > From: jeff saremi <jeffsar...@hotmail.com> > > > > > Sent: Wednesday, May 24, 2017 9:04:11 AM > > > > > To: user@hbase.apache.org > > > > > Subject: What is Dead Region Servers and how to clear them up? > > > > > > > > > > Apparently having dead region servers is so common that a section > of > > > the > > > > > master console is dedicated to that? > > > > > How can we clean this up (preferably in an automated fashion)? Why > > > isn't > > > > > this being done by HBase automatically? > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > >
