Yes, you can run the collector on any host with network access to HDFS. Verify that HADOOP_HOME and HADOOP_CONF_DIR are properly set in chukwa-env.sh. Also verify that you can connect to hadoop from the command line with those environment variables set.
On Tuesday, September 7, 2010, Shreya Chakravarty <[email protected]> wrote: > Hi, > > We want to setup a collector on a non HDFS machine > Is that possible? Will the collector be able to send data to hdfs, we tried > setting the writer.hdfs.filesystem property to point to the hdfs master, but > getting the following error: > > > 2010-09-07 05:53:11,153 INFO main ChukwaConfiguration - chukwaConf is > /home/hadoop/chukwa-0.4.0/conf/ > 2010-09-07 05:53:12,721 INFO main root - initing servletCollector > 2010-09-07 05:53:12,727 INFO main PipelineStageWriter - using pipelined > writers, pipe length is 2 > 2010-09-07 05:53:12,751 INFO Thread-9 SocketTeeWriter - listen thread started > 2010-09-07 05:53:12,755 INFO main SeqFileWriter - rotateInterval is 300000 > 2010-09-07 05:53:12,755 INFO main SeqFileWriter - outputDir is /chukwa/logs/ > 2010-09-07 05:53:12,755 INFO main SeqFileWriter - fsname is > iaas-1-0-16-3e-b7-d1-29:54310 > 2010-09-07 05:53:12,756 INFO main SeqFileWriter - filesystem type from > core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem > 2010-09-07 05:53:12,860 ERROR main SeqFileWriter - can't connect to HDFS, > trying default file system instead (likely to be local) > java.io.IOException: No FileSystem for scheme: iaas-1-0-16-3e-b7-d1-29 > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) > at > org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123) > at > org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88) > at > org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112) > at > org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440) > at > org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685) > at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) > at > org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) > at org.mortbay.jetty.Server.doStart(Server.java:224) > at > org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) > at > org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121) > > Regards > Shreya > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is the > property of Persistent Systems Ltd. It is intended only for the use of the > individual or entity to which it is addressed. If you are not the intended > recipient, you are not authorized to read, retain, copy, print, distribute or > use this message. If you have received this communication in error, please > notify the sender and delete all copies of this message. Persistent Systems > Ltd. does not accept any liability for virus infected mails. >
