Ok, I see now. If you want to collect logs what I would do is a directory spooling source on the namenode instead of messing with hdfs, sounds too risky.
On 1 December 2015 at 11:04, yogendra reddy <[email protected]> wrote: > My usecase is to copy hadoop service logs and that is why I'm adding flume > log4j appender to hadoop log4j properties. > > # Define the root logger to the system property "hadoop.root.logger". > log4j.rootLogger=${hadoop.root.logger}, *flume* > > #Flume Appender > log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender > log4j.appender.flume.Hostname = localhost > log4j.appender.flume.Port = 41414 > > now, for this to work I need to add the flume log4j appender related > libraries to hadoop classpath right? If not this will throw > NoSuchClass found expection org.apache.flume.clients. > log4jappender.Log4jAppender. > > > On Tue, Dec 1, 2015 at 4:24 PM, Gonzalo Herreros <[email protected]> > wrote: > >> That doesn't sound right: >> -Flume should use it's own log4j.properties in the conf directory >> -Never update the hdfs libs to add stuff you need for Flume, each product >> has it's own library directories >> >> Regards, >> Gonzalo >> >> >> On 1 December 2015 at 10:48, yogendra reddy <[email protected]> >> wrote: >> >>> I din't follow. I'm adding flume libraries to hadoop classpath i.e >>> hadoop-hdfs lib folder and this is causing the issue. I need these jars to >>> be in hdfs lib as I have added log4j appender to hdfs log4j properties. >>> >>> On Tue, Dec 1, 2015 at 4:09 PM, Gonzalo Herreros <[email protected]> >>> wrote: >>> >>>> Adding a library to Flume shouldn't affect hive or any other tools. >>>> You can add the jar to the lib or plugin.d directories. >>>> >>>> Regards, >>>> Gonzalo >>>> >>>> On 1 December 2015 at 10:13, yogendra reddy <[email protected]> >>>> wrote: >>>> >>>>> update >>>>> >>>>> I ran the flume agent first and then made changes to hadoop log4j >>>>> properties file and after the restart this started working fine. but now >>>>> Hive service is not coming up because of avro-ipc jar that I had to add to >>>>> hadoop-hdfs lib to get flume log4j appender working >>>>> >>>>> I would like to know if anybody here has used flume to copy Hadoop >>>>> daemon/service logs? >>>>> >>>>> Thanks, >>>>> Yogendra >>>>> >>>>> On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I'm trying to configure flume to write hadoop service logs to a >>>>>> common sink. >>>>>> >>>>>> Here's what I have added to hdfs log4j.properties >>>>>> >>>>>> # Define the root logger to the system property "hadoop.root.logger". >>>>>> log4j.rootLogger=${hadoop.root.logger}, flume >>>>>> >>>>>> #Flume Appender >>>>>> log4j.appender.flume = >>>>>> org.apache.flume.clients.log4jappender.Log4jAppender >>>>>> log4j.appender.flume.Hostname = localhost >>>>>> log4j.appender.flume.Port = 41414 >>>>>> >>>>>> and when I run sample pi job I get this error >>>>>> >>>>>> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10 >>>>>> log4j:ERROR Could not find value for key log4j.appender.flume.layout >>>>>> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default >>>>>> maxIOWorkers >>>>>> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host: >>>>>> localhost, port: 41414 }: RPC connection error >>>>>> Exception in thread "main" java.lang.ExceptionInInitializerError >>>>>> at org.apache.hadoop.util.RunJar.run(RunJar.java:200) >>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136) >>>>>> Caused by: org.apache.commons.logging.LogConfigurationException: >>>>>> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' >>>>>> cannot be found or is not useable. >>>>>> at >>>>>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804) >>>>>> at >>>>>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541) >>>>>> at >>>>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) >>>>>> at >>>>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) >>>>>> at >>>>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657) >>>>>> at >>>>>> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44) >>>>>> ... 2 more >>>>>> >>>>>> I have added these jars to hadoop-hdfs lib - >>>>>> >>>>>> avro-ipc-1.7.3.jar >>>>>> flume-avro-source-1.5.2.2.2.7.1-33.jar >>>>>> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar >>>>>> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar >>>>>> flume-ng-sdk-1.5.2.2.2.7.1-33.jar >>>>>> >>>>>> and I do have the commons-logging( commons-logging-1.1.3.jar) and >>>>>> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug this >>>>>> issue? >>>>>> >>>>>> Thanks, >>>>>> Yogendra >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
