I fixed it in 0.20.3 J-D
2010/1/14 stack <[email protected]>: > Its an exception thrown when you have logging set to DEBUG. Its just an > emission to give a sense of where zk setup is during startup. The issue of > nonode for the master is soon resolved after master succeeds writing its > address into zk and startup proceeds. > > You are not the first to finger this exception as possible issue. Its > confusing. Let us try and change this one at least before we cut the new > 0.20.3RC. > > St.Ack > > 2010/1/14 王仲远 <[email protected]> > >> Hi, J-D, >> Thank you for your response and suggestion. I found the problem was >> caused by /etc/hosts in 10.37.248. In the file, the first line is: >> 127.0.0.1 WAMDM1 localhost.localdomain localhost >> and We used the hostname in our regionservers file. >> >> Then I had corrected the hosts file and solved the problems. >> >> However, when I restarted the hbase, I get some exceptions as follow: >> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/master >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:332) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrapper.java:318) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddress(ZooKeeperWrapper.java:250) >> at >> org.apache.hadoop.hbase.master.ZKMasterAddressWatcher.waitForMasterAddressAvailability(ZKMasterAddressWatcher.java:82) >> at >> org.apache.hadoop.hbase.master.HMaster.writeAddressToZooKeeper(HMaster.java:267) >> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:254) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >> >> But it seemed that Hbase still worked well. What's the matter with this >> exception? >> >> Any suggestion? Thank you~~ >> >> >> Zhongyuan >> 2010-01-15 >> >> >> ------------------------------------------------------------- >> From: Jean-Daniel Cryans >> Sent: 2010-01-14 01:40:33 >> To: hbase-user >> CC: >> Subject: Re: Problems with Hbase Restart >> >> It seems your master's region server reported itself as 127.0.0.1 >> Then there's something wrong with 10.37.17.248, you should check this >> machine's region server log. >> J-D >> 2010/1/13 王仲远 <[email protected]>: >> > I have two problems when I use hbase 0.20.2. >> > >> > I used several machines to make a cluster and deployed hadoop and hbase >> on them. I followed the "Getting Started" and installed Hbase 0.20.2. I >> could start hbase sucessfully and created table and inserted data. >> > >> > However, when I accessed the master page: >> http://10.37.17.252:60010/master.jsp , I found the first problem: the >> master node also appear on Region Servers and appear twice. But I didn't set >> the master node as regionserver in conf/regionservers: >> > Region Servers >> > Address Start Code Load >> > 127.0.0.1:60030 1263383321075 requests=0, regions=0, usedHeap=0, >> maxHeap=0 >> > 127.0.0.1:60030 1263383321096 requests=0, regions=0, usedHeap=0, >> maxHeap=0 >> > WAMDM1.ruc.edu.cn:60030 1263383350174 requests=0, regions=0, >> usedHeap=24, maxHeap=991 >> > WAMDM2.ruc.edu.cn:60030 1263383320980 requests=0, regions=1, >> usedHeap=32, maxHeap=991 >> > WAMDM3.ruc.edu.cn:60030 1263383320985 requests=0, regions=1, >> usedHeap=33, maxHeap=991 >> > WAMDM4.ruc.edu.cn:60030 1263383322246 requests=0, regions=0, >> usedHeap=33, maxHeap=991 >> > >> > However, the hbase seemed to work fine. But When I try to restart the >> hbase, I found the second problem: I run bin/stop-hbase.sh and then run bin/ >> start-hbase.sh. But at this time, the hbase go down. When I tried to access >> the master page: http://10.37.17.252:60010/master.jsp , I got the >> following error message: >> > >> > HTTP ERROR: 500 >> > Trying to contact region server null for region , row '', but failed >> after 3 attempts. >> > Exceptions: >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > >> > RequestURI=/master.jsp >> > >> > Caused by: >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to >> contact region server null for region , row '', but failed after 3 attempts. >> > Exceptions: >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433) >> > at >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) >> > at >> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125) >> > at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) >> > at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) >> > at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> > at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> > at >> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> > at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> > at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:324) >> > at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> > at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) >> > at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> > at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> > >> > Powered by Jetty:// >> > >> > >> > The error messages in the master log: >> > 2010-01-13 18:34:04,424 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:04,425 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:04,425 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:06,425 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:06,429 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:06,430 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:06,430 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:08,430 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:08,434 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:08,435 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:08,435 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407) >> > at >> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57) >> > at >> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433) >> > at >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) >> > at >> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125) >> > at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) >> > at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) >> > at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> > at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> > at >> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> > at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> > at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:324) >> > at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> > at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) >> > at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> > at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> > 2010-01-13 18:34:08,451 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:08,451 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:08,451 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:10,451 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:10,456 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:10,465 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:10,465 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:12,465 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:12,469 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:12,470 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:14,474 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:14,475 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:14,475 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:16,475 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:16,480 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:16,480 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:16,480 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:18,480 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:18,485 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:18,486 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:18,486 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407) >> > at >> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57) >> > at >> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433) >> > at >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) >> > at >> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125) >> > at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) >> > at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) >> > at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> > at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> > at >> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> > at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> > at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:324) >> > at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> > at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) >> > at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> > at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> > 2010-01-13 18:34:18,500 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:18,501 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:18,501 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:20,501 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:20,506 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:20,506 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:20,506 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:22,507 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:22,511 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:22,511 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:24,516 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:24,516 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:24,516 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:26,517 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:26,521 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:26,522 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:26,522 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:28,522 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:28,526 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:28,527 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:28,527 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: >> locateRegionInMeta attempt 0 of 3 failed; retrying after sleep of 2000 >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:936) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:581) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:631) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407) >> > at >> org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57) >> > at >> org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:62) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55) >> > at >> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433) >> > at >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) >> > at >> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125) >> > at >> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) >> > at >> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) >> > at >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> > at >> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> > at >> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> > at >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> > at >> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:324) >> > at >> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> > at >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) >> > at >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> > at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> > 2010-01-13 18:34:28,547 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:28,547 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:28,547 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:30,547 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:30,552 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:30,552 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:30,552 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region >> location changed. Sleeping. >> > 2010-01-13 18:34:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:34:32,552 DEBUG >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry >> finding root region. >> > 2010-01-13 18:34:32,557 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode >> /hbase/root-region-server got 10.37.17.248:60020 >> > 2010-01-13 18:34:32,558 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:34:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:34:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:35:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:35:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:35:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:35:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:35:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:36:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:36:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:36:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:36:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:36:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:37:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:37:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:37:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:37:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:38:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:38:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:38:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:38:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:38:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:39:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:39:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:39:32,741 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:39:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > 2010-01-13 18:39:32,742 INFO org.apache.hadoop.hbase.master.BaseScanner: >> All 0 .META. region(s) scanned >> > 2010-01-13 18:40:31,575 INFO >> org.apache.hadoop.hbase.master.ServerManager: 5 region servers, 0 dead, >> average load 0.16666666666666666 >> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.37.17.248:60020, >> regionname: -ROOT-,,0, startKey: <>} >> > 2010-01-13 18:40:32,740 INFO org.apache.hadoop.ipc.HbaseRPC: Server at / >> 10.37.17.248:60020 could not be reached after 1 tries, giving up. >> > 2010-01-13 18:40:32,741 WARN org.apache.hadoop.hbase.master.BaseScanner: >> Scan ROOT region >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting >> up proxy to /10.37.17.248:60020 after attempts=1 >> > at >> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:424) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:865) >> > at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:881) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:156) >> > at >> org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54) >> > at >> org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79) >> > at >> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:136) >> > at org.apache.hadoop.hbase.Chore.run(Chore.java:68) >> > ..... >> > >> > >> > Can somebody help me. Thank you. >> > ------------------ >> > Zhongyuan >> > 2010-01-13 >> > >> > >> > >> >
