I had checked these before posting, the problem was in the “initial_adaptors”
file, I had copied
“add FileTailer FooData /tmp/foo 0” from the Chukwa agent documentation, it
didn’t work. I had
to change to “add filetailer.FileTailingAdaptor FooData /tmp/foo 0” to make it
work. I found the
exception in “agent.log” and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this
exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread
ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the
stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at
org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at
org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at
org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at
org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at
org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at
org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread
ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper -
The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration -
chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration -
chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean -
java.lang.NullPointerException
at
org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at
org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at
org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at
org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at
org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at
org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at
org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: 邵赛赛 [mailto:[email protected]]
Sent: Monday, September 17, 2012 6:28 PM
To: [email protected]
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught
when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind
correct port.
2. maybe port 8081 has been occupied by other process, so please check the
collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <[email protected]>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning,
any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext -
Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME:
[/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR:
[/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME:
[chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir:
[/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE:
[true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS:
[10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period
is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in
/opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id =
adaptor_93df4746476c9a4b624f6755b122f9dc
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id =
adaptor_79203ebcc495e11b747254e660380620
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id =
adaptor_e2b39549cac6e966126df286a79ae6ad
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id =
adaptor_c79bf882974a14286cffe29d3d4cf0d6
function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id =
adaptor_c6fd1e5021ad8023477ef0b56a24a29f
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in
/opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener -
AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to
9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port
9093
2012-09-17 16:00:29,713 INFO main log - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started
[email protected]:9090
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent
interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector
started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up
collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors
from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since
last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1
chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP
post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post
exception on http://localhost:8081/: java.net.ConnectException: Connection
refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more
collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught
java.io.EOFException closing conneciton.
