Thanks Hari , As you told it is working. Im able to see the human redable format.
The following is the output: 20000 UTF8 1344001730307 com.cisco.flume.FlumeTest Sample info message 30000 UTF8 1344001730432 com.cisco.flume.FlumeTest Sample warn message 40000 UTF8 1344001730463 com.cisco.flume.FlumeTest Sample error message 50000 UTF8 1344001730479 com.cisco.flume.FlumeTest Sample fatal message i want the following logging pattern *%d \=\=\=\= thread\: %t \=\=\=\= %-8p>%n%c.%M() \=> %x (line\: %L)%n\t%m%n%n* *what changes required , to get this pattern work* Thanks JP On Mon, Jul 30, 2012 at 11:52 PM, Hari Shreedharan < [email protected]> wrote: > You are using the AvroEventSerializer. This formats the event into Avro > format specified by > org.apache.flume.serialization.FlumeEventAvroEventSerializer, > which is why it looks like garbage, while it is not. Your app should be > written to read and understand the Avro format. If you need it to human > readable, you will need to write your own serializer, perhaps by extending > the BodyTextEventSerializer. > > Thanks > Hari > > -- > Hari Shreedharan > > On Monday, July 30, 2012 at 9:34 AM, JP wrote: > > Thanks Hari , > > i got little progress. > > But im getting garbage values. > > this is my configurations: > > *flume-conf.properties* > --------------------------------------- > agent2.sources = seqGenSrc > agent2.channels = memoryChannel > agent2.sinks = loggerSink > > agent2.sources.seqGenSrc.type = avro > agent2.sources.seqGenSrc.bind=localhost > agent2.sources.seqGenSrc.port=41414 > > agent2.channels.memoryChannel.type = memory > agent2.channels.memoryChannel.capacity = 1000000 > agent2.channels.memoryChannel.transactionCapacity = 1000000 > agent2.channels.memoryChannel.keep-alive = 30 > > agent2.sources.seqGenSrc.channels = memoryChannel > > agent2.sinks.loggerSink.type = hdfs > agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs > agent2.sinks.loggerSink.hdfs.fileType = DataStream > agent2.sinks.loggerSink.channel = memoryChannel > agent2.sinks.loggerSink.serializer = avro_event > agent2.sinks.loggerSink.serializer.compressionCodec = snappy > agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000 > agent2.channels.memoryChannel.type = memory > > > log4j.properties > > ------------------------------------------------------------------------------ > log4j.rootLogger=INFO, CA, flume > > log4j.appender.CA=org.apache.log4j.ConsoleAppender > > log4j.appender.CA.layout=org.apache.log4j.PatternLayout > log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n > > log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender > log4j.appender.flume.Hostname = localhost > log4j.appender.flume.Port = 41414 > > > and my output: > ------------------------ > Obj avro.codec null avro.schema� > {"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�| > ��(r5��q ��nl � 8flume.client.log4j.log.level > 40000Fflume.client.log4j.message.encoding > UTF88flume.client.log4j.timestamp > 1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample > error message| ��(r5��q ��nl � 8flume.client.log4j.log.level > 50000Fflume.client.log4j.message.encoding > UTF88flume.client.log4j.timestamp > 1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample > fatal message| ��(r5��q ��nl � 8flume.client.log4j.log.level > 20000Fflume.client.log4j.message.encoding > UTF88flume.client.log4j.timestamp > > > Please let me know, if im in the wrong path. > > Please suggest me to get custom logging pattern (for example like in log4j) > > > Thanks > JP > > On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan < > [email protected]> wrote: > > + user@ > > Thamatam, > > The Log4J appender adds the date, log level and logger name to the flume > event headers and the text of the log event to the flume event body. The > reason the log level and time are missing is that these are in the headers > and the text serializer does not serialize the headers. > > To write to a file or HDFS, please use a Serializer together with the > RollingFileSink or HDFSEventSink. Please take a look at the plain text > serializer or Avro serializer to understand this better. > > Thanks, > Hari > > -- > Hari Shreedharan > > On Saturday, July 28, 2012 at 5:47 PM, thamatam Jayaprakash wrote: > > Hi Hari, > > > Actually im unable to send to this mail to the user and dev group so, im > mailing to you . > > Could you pls point me where im going wrong. > *Please suggest me which log appender need to use for custom logging > pattern and appender.* > > Im working on Flume 1.1.0 and 1.2.0 . We are not able set log pattern and > We are using log4jappender > log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender > > but we are getting plain test > > *Example if i log following mssages :* > > 17:42:55,928 INFO SimpleJdbcServlet:69 - doGet of SimpleJdbcServlet > ended... > 17:43:03,489 INFO HelloServlet:29 - HelloServlet of doGet started... > 17:43:03,489 INFO HelloServlet:33 - > Hello from Simple Servlet > 17:43:03,489 INFO HelloServlet:35 - HelloServlet of doGet end... > 17:47:46,000 INFO HelloServlet:29 - HelloServlet of doGet started... > 17:47:46,001 INFO HelloServlet:33 - > Hello from Simple Servlet > 17:47:46,001 INFO HelloServlet:35 - HelloServlet of doGet end... > > *Using Flume in Hadoop im getting only the following logs:* > > doGet of SimpleJdbcServlet ended... > HelloServlet of doGet started... > > HelloServlet of doGet end... > HelloServlet of doGet started... > > * > > Thanks in advance. > * > -- > JP > > > > -- > Jayaprakash > > > > > > -- > JP > > > -- JP -- JP
