Glad it works. Check point file should be located in CHUKWA_LOG_DIR. regards, Eric
On Oct 27, 2011, at 6:28 PM, AD wrote: > that worked thanks. Out of curiosity where is this persisted (i restarted > and it came back apparently). > > > On Thu, Oct 27, 2011 at 4:11 PM, Eric Yang <[email protected]> wrote: > Agent check point file may be storing the multiple adaptors, make sure you > use port 9093 to stop the existing running adaptors. > > regards, > Eric > > On Oct 27, 2011, at 12:00 PM, AD wrote: > > > Hmm, even after removing i am still seeing 2 tails being started with lsof. > > I am only using 1 filetailer.CharFileTailingAdaptorUTF8 in > > initial_adaptors. > > > > > > > > On Thu, Oct 27, 2011 at 1:07 PM, AD <[email protected]> wrote: > > argh. yes. I guess # is not a comment in the initial_adaptors file :-) i > > commented out the old filetailer.FileTailingAdaptor for the new > > filetailer.CharFileTailingAdaptorUTF8 > > > > anyway thanks for the help. > > > > > > > > On Thu, Oct 27, 2011 at 12:19 PM, Eric Yang <[email protected]> wrote: > > Do you have two data stream that are tailing the same file? To verify: > > > > telnet agent-host 9093 > > list > > > > Make sure you are not streaming the same data twice. > > > > regards, > > Eric > > > > On Oct 27, 2011, at 7:25 AM, AD wrote: > > > > > I migrated to the filetailer.CharFileTailingAdaptorUTF8 which ensured > > > newline breaks but i am now getting 2 of every entry. Put the below > > > debug code into the demux parser, any ideas why i would be seeing 2 of > > > every event in the log? > > > > > > throws Throwable { > > > try { > > > String lines[] = recordEntry.split("\\r?\\n"); > > > FileWriter out = new FileWriter("/tmp/demux.test",true); > > > PrintWriter p = new PrintWriter(out); > > > for(int i = 0; i < lines.length; i++) { > > > log.warn("*** TRYING TO PARSE **** " + lines[i]); > > > p.println(lines[i]); > > > } > > > p.close(); > > > > > > On Thu, Oct 27, 2011 at 6:39 AM, AD <[email protected]> wrote: > > > yep that was it thanks. For the "recordEntry" variable, is there any way > > > to guarantee how this is structured? I am testing tailing a file and i > > > notice recordEntry in the parser is around 20-30 lines from the logfile, > > > is this expected? can we safely assume newline termination and just loop > > > through? I was expecting 1 recordEntry for each line of the logfile but > > > i think this has to do with it being a chukwaRecord not a log entry. > > > > > > > > > On Thu, Oct 27, 2011 at 1:25 AM, Eric Yang <[email protected]> wrote: > > > Do you have both chukwa-core.jar and chukwa-core-0.4.0.jar? Run jar tf > > > chukwa-core-0.4.0.jar | grep TsProcessor2, does chukwa-core-0.4.0.jar > > > file have TsProcessor2.class? > > > > > > regards, > > > Eric > > > > > > On Oct 26, 2011, at 7:59 PM, AD wrote: > > > > > > > Hey Eric, > > > > > > > > So as a test, i copied TsProcessor.java to TsProcessor2.java. I > > > > changed the references for TsProcessor to be TsProcessor2 and updated > > > > my chukwa-demux.conf to be > > > > > > > > <property> > > > > <name>TsProcessor</name> > > > > > > > > <value>org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2</value> > > > > <description>Parser class for </description> > > > > </property> > > > > > > > > I then ran ant in the root and copied build/collector-0.4.0.war and > > > > build/chukwa-core.jar to the root and started collector and demux. > > > > > > > > I am now getting the following errors, any ideas ? > > > > > > > > org.apache.hadoop.chukwa.extraction.demux.processor.mapper.UnknownRecordTypeException: > > > > Unknown > > > > parserClass:org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 > > > > > > > > Caused by: java.lang.ClassNotFoundException: > > > > org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor2 > > > > > > > > On Wed, Oct 26, 2011 at 1:00 PM, Eric Yang <[email protected]> wrote: > > > > Yes, write a mapper class which extends > > > > org/apache/hadoop/chukwa/extraction/demux/processor/mapper/AbstractProcessor.java. > > > > There are several extraction class in > > > > org/apache/hadoop/chukwa/extraction/demux/processor/mapper as examples. > > > > Once the extraction class is written, configure chukwa-demux-conf.xml > > > > to map the data type to the new extraction class. > > > > > > > > regards, > > > > Eric > > > > > > > > On Oct 26, 2011, at 5:48 AM, AD wrote: > > > > > > > > > interesting. So demux is now part of the collector and does not need > > > > > to be run as another job? > > > > > > > > > > Since demux is doing basic ETL into hbase, is there a way to actually > > > > > parse the fields of the log record and insert those into sep fields > > > > > into Hbase to run mapreduce there (instead of the whole body as a > > > > > field is what I think is happening) > > > > > > > > > > On Wed, Oct 26, 2011 at 12:59 AM, SALAMI Patrick > > > > > <[email protected]> wrote: > > > > > Ok, thanks! > > > > > > > > > > -----Original Message----- > > > > > From: Eric Yang [mailto:[email protected]] On Behalf Of Eric Yang > > > > > Sent: Tuesday, October 25, 2011 6:19 PM > > > > > To: [email protected] > > > > > Subject: Re: Chukwa trunk does not write to HBase > > > > > > > > > > Demuxer for Chukwa 0.4 is a mapreduce job acting as ETL process to > > > > > convert the data into semi structure format for further processing. > > > > > For Chukwa trunk with HBase, demux is running as part of collector > > > > > for ETL. Hence, there is no need to run demux process. > > > > > > > > > > regards, > > > > > Eric > > > > > > > > > > On Oct 25, 2011, at 3:49 PM, SALAMI Patrick wrote: > > > > > > > > > > > Also, while we are chatting, I was hoping to understand the role of > > > > > > the demuxer. I am assuming that HICC pulls all of its data from > > > > > > HBase. If using HBase, is it still necessary to run the demuxer? I > > > > > > didn't see any mention of it in the latest quick start guide. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Patrick > > > > > > > > > > > > -----Original Message----- > > > > > > From: Eric Yang [mailto:[email protected]] On Behalf Of Eric > > > > > > Yang > > > > > > Sent: Tuesday, October 25, 2011 2:45 PM > > > > > > To: [email protected] > > > > > > Subject: Re: Chukwa trunk does not write to HBase > > > > > > > > > > > > Hadoop trunk will require a different configuration than the one > > > > > > described in the Quick_Start guide. > > > > > > > > > > > > 1. Apply this patch: HADOOP-7436, and rebuild hadoop. > > > > > > 2. Copy hadoop-metrics2.properties enclosed in this message to > > > > > > HADOOP_CONF_DIR > > > > > > > > > > > > Restart hadoop. You might need to match the HBase table schema > > > > > > with the metrics emitted by Hadoop metrics 2 framework. > > > > > > Hope this helps. > > > > > > > > > > > > > > > > > > This message contains confidential information and is intended only > > > > > > for the individual(s) named. If you are not the named addressee you > > > > > > should not disseminate, distribute or copy this e-mail. Please > > > > > > notify the sender immediately by e-mail if you have received this > > > > > > e-mail by mistake and delete this e-mail from your system. > > > > > > > > > > > > > > > This message contains confidential information and is intended only > > > > > for the individual(s) named. If you are not the named addressee you > > > > > should not disseminate, distribute or copy this e-mail. Please notify > > > > > the sender immediately by e-mail if you have received this e-mail by > > > > > mistake and delete this e-mail from your system. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
