Can you do the following ? run hbck command to see if any inconsistency is reported check region server log on test-rs3 to see if hbase:meta is successfully opened
Cheers On Fri, Sep 4, 2015 at 8:20 AM, Akmal Abbasov <akmal.abba...@icloud.com> wrote: > Hi Ted, > > No, I’ve restarted master node and now there is no exceptions, but still > webui is not responding > 2015-09-04 14:54:57,926 DEBUG [FifoRpcScheduler.handler1-thread-26] > master.AssignmentManager: based on AM, current > region=hbase:meta,,1.1588230740 is on server=test-rs3,60020,1441378286999 > server being checked: test-rs5,60020,1441261824016 > What else could be wrong? > > Thanks. > > > On 03 Sep 2015, at 16:20, Ted Yu <yuzhih...@gmail.com> wrote: > > Did the exception lead to hbase:meta assignment failure ? > > Please check region server log on 10.10.8.53 <http://10.10.8.53:60000/> > > On Thu, Sep 3, 2015 at 7:15 AM, Akmal Abbasov <akmal.abba...@icloud.com> > wrote: > >> Hi Ted, >> I’m getting a lot of >> 2015-09-03 14:08:20,651 WARN [1865504699@qtp-838397579-13] >> client.HConnectionManager$HConnectionImplementation: Checking master >> connection >> com.google.protobuf.ServiceException: java.net.SocketTimeoutException: >> Call to test-m/10.10.8.53:60000 failed because >> java.net.SocketTimeoutException: 60000 millis timeout while waiting for >> channel to be ready for read. ch : >> java.nio.channels.SocketChannel[connected local=/10.10.8.53:41243 >> remote=test-m/10.10.8.53:60000] >> at >> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678) >> at >> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) >> at >> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:42561) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(HConnectionManager.java:1508) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(HConnectionManager.java:2146) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1859) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3320) >> at >> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:113) >> at >> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3347) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.listTableDescriptorsByNamespace(HBaseAdmin.java:2281) >> at >> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.__jamon_innerUnit__catalogTables(MasterStatusTmplImpl.java:465) >> at >> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:274) >> at >> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:386) >> at >> org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:376) >> at >> org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:95) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:735) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >> at >> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) >> at >> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1122) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at >> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) >> at >> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) >> at java.io.FilterInputStream.read(FilterInputStream.java:133) >> at >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:558) >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:254) >> at java.io.DataInputStream.readInt(DataInputStream.java:387) >> at >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076) >> at >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727) >> >> Thanks >> >> > On 03 Sep 2015, at 15:50, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> > Can you log into the master node and check master log ? >> > >> > Pastebin snippet of master log if you need more help. >> > >> > Cheers >> > >> > >> >> On Sep 3, 2015, at 6:24 AM, Akmal Abbasov <akmal.abba...@icloud.com> >> wrote: >> >> >> >> Hi, >> >> I’m having problems with accessing HBase master webui. when I try to >> access it, it returns redirect page /master-status, and then it is freezes. >> >> I’ve tried to wait for a long time, but still it didn’t work. >> >> I am using hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1 >> >> Any suggestions? >> >> Thanks. >> >> > >
