Hello AD, I am sorry for leaving the discussion in between, I was busy with my exams.Now I am back and need your help.I started with what you have suggested and read about the concepts (agents and collectors etc). Now I am trying to collect the data from a file on the local file system and trying to dump it into the HDFS and I am running the agent and collector on the same machine.I was able to start the agent but when I tried to start the collector I got the following line at the terminal and the terminal got stuck there itself - tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2011-11-10 18:42:09.544::INFO: jetty-6.1.11
When I issued jps, I could see only the agent. Regards, Mohammad Tariq On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[email protected]> wrote: > One difference between Chukwa and Flume is that in Chukwa you configure each > component individually to know how to handle different datatypes. This makes > it fairly straight-forward to set up, but can be painful to manage for very > large installs (so I've heard). Flume centralized the configs of "flows" > which helps when managing larger installs, at the cost of additional > complexity. > > On Fri, Nov 4, 2011 at 7:02 AM, AD <[email protected]> wrote: >> >> Are you running the agent and collector on the same machine? Off the cuff >> its something like >> 1 - set your CHUKWA_HOME env variable >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the vars look >> good >> 3 - edit conf/initial_adaptors with the adaptors you need. note here that >> the dataType >> 4 - edit conf/collectors and add the hostname/ip of the collectors >> 5 - edit conf/agents and add ip/hostname of the agents >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux processor >> for your dataType (name from initial_adaptors) >> On agent node run bin/chukwa agent , tail ${CHUKWA_LOGS}/agent.log >> On collector node run bin/chukwa collector , tail >> ${CHUKWA_LOGS]/collector.log >> On collector node run bin/chukwa Demux , tail ${CHUKWA_LOGS}/Demux.log >> >> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[email protected]> wrote: >>> >>> Hello AD, >>> Yes, I have read the wiki and right now I am trying to set up a >>> Chukwa cluster using the wiki itself. Are there any other links you >>> are aware of that can help me learn and use Chukwa in a better >>> fashion?? I am trying to use Chukwa for data in "The Wellsite >>> Information Transfer Standard Markup Language (WITSML)" format. >>> >>> Regards, >>> Mohammad Tariq >>> >>> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[email protected]> wrote: >>> > >>> > I think it helps to read he wiki and understand the basic concepts, >>> > like agents and collectors. What kind of data do you want to parse? I >>> > have >>> > used both Flume and Chukwa myself and can say for sure Chukwa is miles >>> > easier to work with and configure. Flume may have some more advanced >>> > features, but so far i found Chukwa both easy to extend from a development >>> > standpoint, and much easier to configure. >>> > Have you read the wiki pages ? >>> > >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[email protected]> >>> > wrote: >>> >> >>> >> Hello list, >>> >> I am totally new to Chukwa and I have to learn and start using it as >>> >> soon soon as possible as a part of my new project. Could any one let me >>> >> know >>> >> how to begin with?? It would be a great favor to me. I know Flume and >>> >> have >>> >> used it already.Also I would like to know how different is Chukwa from >>> >> Flume?? >>> >> Many thanks in advance. >>> >> >>> >> Regards, >>> >> Mohammad Tariq >>> > >> > >
