If I understand the issue correctly, restarting the master should solve the problem.
On Wed, Mar 4, 2015 at 5:55 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Please see HBASE-13067 Fix caching of stubs to allow IP address changes of > restarted remote servers > > Cheers > > On Tue, Mar 3, 2015 at 8:26 PM, Sandeep L <sandeepvre...@outlook.com> > wrote: > > > Hi nkeywal, > > While trying to get more details about this issue I got to know that > > HMaster is trying to connect to wrong IP Address. > > Here is exact issue: > > Due to some unavoidable reason we are forced to change IP Address of > > regionsserver & then updated new IP Address in /etc/hosts file across all > > HBase servers. I started RegionServer from master with start-hbase.sh > > scripts & jps output in regionserver shows it's(regionserver process) up > > and running. > > But when running hbase balancer HMaster is trying to connect to old IP > > Address instead of new IP Address. > > One more thing here is when I checked regionserver status on 60010 port > > its showing as up and running. > > Thanks,Sandeep. > > > > > From: nkey...@gmail.com > > > Date: Tue, 3 Mar 2015 19:01:01 +0100 > > > Subject: Re: Where is HBase failed servers list stored > > > To: user@hbase.apache.org > > > > > > It's in local memory. When HBase cannot connect to a server, it puts it > > > into the "failedServerList" for 2 seconds. This is to avoid having all > > the > > > threads going into a potentially long socket timeout. Are you sure that > > you > > > can connect from the master to this machine/port? > > > > > > You can change the time it stays in the list with > > > hbase.ipc.client.failed.servers.expiry (in milliseconds), but it should > > not > > > help. > > > > > > You should have another exception before this one in the logs (the one > > that > > > initially put this region server in this failedServerList). > > > > > > On Tue, Mar 3, 2015 at 12:08 PM, Sandeep L <sandeepvre...@outlook.com> > > > wrote: > > > > > > > Hi, > > > > While trying to run hbase balancer I am getting error message as > "This > > > > server is in the failed servers list".Due to this cluster is not > > getting > > > > balanced. > > > > Even though regionserver is up and running hmaster is unable to > > connect to > > > > it. > > > > The odd thing here is hmaster is able to start regionserver and it is > > > > detected as up and running but unable to assign regions. > > > > Can some one suggest any solution for this. > > > > Following is full stack > > > > trace:org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: > This > > > > server is in the failed servers list: host1/192.168.2.20:60020 at > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) > > > > at > > > > > > org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) > > > > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) > > at > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) > > > > at > > > > > > > org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) > > > > at > > > > > > > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964) > > > > at > > > > > > > org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550) > > > > at > > > > > > > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:999) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447) > > > > at > > > > > > > org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260) > > > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > > at > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > > at java.lang.Thread.run(Thread.java:745) > > > > Thanks,Sandeep. > > > > >
