I din't follow. I'm adding flume libraries to hadoop classpath i.e hadoop-hdfs lib folder and this is causing the issue. I need these jars to be in hdfs lib as I have added log4j appender to hdfs log4j properties.
On Tue, Dec 1, 2015 at 4:09 PM, Gonzalo Herreros <[email protected]> wrote: > Adding a library to Flume shouldn't affect hive or any other tools. > You can add the jar to the lib or plugin.d directories. > > Regards, > Gonzalo > > On 1 December 2015 at 10:13, yogendra reddy <[email protected]> wrote: > >> update >> >> I ran the flume agent first and then made changes to hadoop log4j >> properties file and after the restart this started working fine. but now >> Hive service is not coming up because of avro-ipc jar that I had to add to >> hadoop-hdfs lib to get flume log4j appender working >> >> I would like to know if anybody here has used flume to copy Hadoop >> daemon/service logs? >> >> Thanks, >> Yogendra >> >> On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <[email protected]> >> wrote: >> >>> Hi All, >>> >>> I'm trying to configure flume to write hadoop service logs to a common >>> sink. >>> >>> Here's what I have added to hdfs log4j.properties >>> >>> # Define the root logger to the system property "hadoop.root.logger". >>> log4j.rootLogger=${hadoop.root.logger}, flume >>> >>> #Flume Appender >>> log4j.appender.flume = >>> org.apache.flume.clients.log4jappender.Log4jAppender >>> log4j.appender.flume.Hostname = localhost >>> log4j.appender.flume.Port = 41414 >>> >>> and when I run sample pi job I get this error >>> >>> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10 >>> log4j:ERROR Could not find value for key log4j.appender.flume.layout >>> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default maxIOWorkers >>> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host: >>> localhost, port: 41414 }: RPC connection error >>> Exception in thread "main" java.lang.ExceptionInInitializerError >>> at org.apache.hadoop.util.RunJar.run(RunJar.java:200) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136) >>> Caused by: org.apache.commons.logging.LogConfigurationException: >>> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger' >>> cannot be found or is not useable. >>> at >>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804) >>> at >>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541) >>> at >>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) >>> at >>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) >>> at >>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657) >>> at >>> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44) >>> ... 2 more >>> >>> I have added these jars to hadoop-hdfs lib - >>> >>> avro-ipc-1.7.3.jar >>> flume-avro-source-1.5.2.2.2.7.1-33.jar >>> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar >>> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar >>> flume-ng-sdk-1.5.2.2.2.7.1-33.jar >>> >>> and I do have the commons-logging( commons-logging-1.1.3.jar) and >>> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug this >>> issue? >>> >>> Thanks, >>> Yogendra >>> >>> >>> >> >
