Re: What is Dead Region Servers and how to clear them up?

2017-05-27 Thread Ted Yu
The involvement of Yarn can explain why you observed relatively more dead servers (compared to traditional deployment). Suppose in first run, Yarn allocates containers for region servers on a set of nodes. Subsequently, Yarn may choose nodes (for the same number of servers) which are not exactly t

Re: What is Dead Region Servers and how to clear them up?

2017-05-27 Thread jeff saremi
Yes. we don't have fixed servers with the exceptions of ZK machines. We have 3 yarn jobs one for each of master, region, and thrift servers each launched separately with different number of nodes. I hope that's not what is causing problems. From: Ted Yu Sent: S

Re: What is Dead Region Servers and how to clear them up?

2017-05-27 Thread Ted Yu
Jeff: bq. We run our cluster on Yarn and upon restarting jobs in Yarn Can you clarify a bit more - are you running hbase processes inside Yarn container ? Cheers On Sat, May 27, 2017 at 10:58 AM, jeff saremi wrote: > Thanks @Yu Li > > You are absolutely correct. Dead R

Re: What is Dead Region Servers and how to clear them up?

2017-05-27 Thread jeff saremi
Thanks @Yu Li You are absolutely correct. Dead RS's will happen regardless. My issue with this is more "psychological". If I have done everything needed to be done to ensure that RSs are running fine and regions are assigned and such and hbck reports are consistent then