This is the output on my terminal when i issue
"prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" -
prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run
program "/usr/bin/iostat": java.io.IOException: error=2, No such file
or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at
org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at
org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
java.io.IOException: Cannot run program "/usr/bin/sar":
java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:328)
at
org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66)
at
org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: java.io.IOException: error=2, No such
file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 7 more
and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa
collector" i get this -
prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO:
Logging to STDERR via org.mortbay.log.StdErrLog
2011-11-11 01:04:25.894::INFO: jetty-6.1.11
Regards,
Mohammad Tariq
On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[email protected]> wrote:
> Hello AD,
> this my initial_adaptors file -
>
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat
> 60 /usr/bin/iostat -x -k 55 2 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60
> /bin/df -l 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60
> /usr/bin/sar -q -r -n ALL 55 0
> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60
> /usr/bin/top -b -n 1 -c 0
>
> and this is the collectors file -
> localhost
>
> I am trying to run both agent and collector on the same machine.
>
> The collector.log file looks like this -
> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf is
> /home/prashant/chukwa-0.4.0/bin/../conf
> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector
> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using
> pipelined writers, pipe length is 2
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is 300000
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is /chukwa/logs/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is
> hdfs://localhost:9000/
> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type from
> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
> 2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen thread started
> 2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to
> HDFS, trying default file system instead (likely to be local)
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
> at
> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at
> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216)
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409)
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395)
> at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
> at
> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123)
> at
> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at
> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at
> org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.configuration.Configuration
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 29 more
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Nov 10, 2011 at 8:30 PM, AD <[email protected]> wrote:
>> what happens when you tail the collector.log file?
>> can you share your initial_adaptors and collectors file in conf ?
>>
>> On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <[email protected]> wrote:
>>>
>>> Hello AD,
>>> I am sorry for leaving the discussion in between, I was busy with
>>> my exams.Now I am back and need your help.I started with what you have
>>> suggested and read about the concepts (agents and collectors etc). Now
>>> I am trying to collect the data from a file on the local file system
>>> and trying to dump it into the HDFS and I am running the agent and
>>> collector on the same machine.I was able to start the agent but when I
>>> tried to start the collector I got the following line at the terminal
>>> and the terminal got stuck there itself -
>>> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging
>>> to STDERR via org.mortbay.log.StdErrLog
>>> 2011-11-10 18:42:09.544::INFO: jetty-6.1.11
>>>
>>> When I issued jps, I could see only the agent.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>>
>>> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[email protected]> wrote:
>>> > One difference between Chukwa and Flume is that in Chukwa you configure
>>> > each
>>> > component individually to know how to handle different datatypes. This
>>> > makes
>>> > it fairly straight-forward to set up, but can be painful to manage for
>>> > very
>>> > large installs (so I've heard). Flume centralized the configs of "flows"
>>> > which helps when managing larger installs, at the cost of additional
>>> > complexity.
>>> >
>>> > On Fri, Nov 4, 2011 at 7:02 AM, AD <[email protected]> wrote:
>>> >>
>>> >> Are you running the agent and collector on the same machine? Off the
>>> >> cuff
>>> >> its something like
>>> >> 1 - set your CHUKWA_HOME env variable
>>> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars
>>> >> look
>>> >> good
>>> >> 3 - edit conf/initial_adaptors with the adaptors you need. note here
>>> >> that
>>> >> the dataType
>>> >> 4 - edit conf/collectors and add the hostname/ip of the collectors
>>> >> 5 - edit conf/agents and add ip/hostname of the agents
>>> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux
>>> >> processor
>>> >> for your dataType (name from initial_adaptors)
>>> >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log
>>> >> On collector node run bin/chukwa collector , tail
>>> >> ${CHUKWA_LOGS]/collector.log
>>> >> On collector node run bin/chukwa Demux , tail
>>> >> ${CHUKWA_LOGS}/Demux.log
>>> >>
>>> >>
>>> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[email protected]>
>>> >> wrote:
>>> >>>
>>> >>> Hello AD,
>>> >>> Yes, I have read the wiki and right now I am trying to set up a
>>> >>> Chukwa cluster using the wiki itself. Are there any other links you
>>> >>> are aware of that can help me learn and use Chukwa in a better
>>> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite
>>> >>> Information Transfer Standard Markup Language (WITSML)" format.
>>> >>>
>>> >>> Regards,
>>> >>> Mohammad Tariq
>>> >>>
>>> >>>
>>> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[email protected]> wrote:
>>> >>> >
>>> >>> > I think it helps to read he wiki and understand the basic concepts,
>>> >>> > like agents and collectors. What kind of data do you want to parse?
>>> >>> > I have
>>> >>> > used both Flume and Chukwa myself and can say for sure Chukwa is
>>> >>> > miles
>>> >>> > easier to work with and configure. Flume may have some more
>>> >>> > advanced
>>> >>> > features, but so far i found Chukwa both easy to extend from a
>>> >>> > development
>>> >>> > standpoint, and much easier to configure.
>>> >>> > Have you read the wiki pages ?
>>> >>> >
>>> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[email protected]>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Hello list,
>>> >>> >> I am totally new to Chukwa and I have to learn and start using it
>>> >>> >> as
>>> >>> >> soon soon as possible as a part of my new project. Could any one
>>> >>> >> let me know
>>> >>> >> how to begin with?? It would be a great favor to me. I know Flume
>>> >>> >> and have
>>> >>> >> used it already.Also I would like to know how different is Chukwa
>>> >>> >> from
>>> >>> >> Flume??
>>> >>> >> Many thanks in advance.
>>> >>> >>
>>> >>> >> Regards,
>>> >>> >> Mohammad Tariq
>>> >>> >
>>> >>
>>> >
>>> >
>>
>>
>
