Hi Meena, Chukwa 0.5.0 has HBase enabled by default. You can modify it to stream to HDFS by following the instructions on this page:
http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html regards, Eric On Wed, Apr 4, 2012 at 1:03 AM, meena k.s <[email protected]> wrote: > Hi Eric, > > Thanks for the quick reply.I have tried to install chukwa 0.5 accordingly. > but there seems to be a problem while starting chukwa-collector. In the log > files i could see something related to hbase. I haven't installed any hbase > in my cluster. Therefore I commented out the parameters related to hbase in > my chukwa-collector-conf.xml. Is it necessary to have a hbase installation > for chukwa 0.5 to work. > > Thanks, > Meena > > On Tue, Apr 3, 2012 at 9:36 PM, Eric Yang <[email protected]> wrote: >> >> Hi Meena, >> >> Hadoop 0.20.205 and Chukwa 0.4 is a mismatch. Hadoop 0.20.20x has >> been modified to use MetricsSink instead of MentricsContext. >> Therefore, you will not see Hadoop metrics by using Chukwa 0.4. You >> will need Chukwa 0.5 to monitor Hadoop 0.20.205+. In addition, you >> should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor >> Hadoop. Hope this helps. >> >> regards, >> Eric >> >> On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <[email protected]> wrote: >> > Hi, >> > >> > >> > >> > We want to use chukwa for hadoop log collection. We are running hive >> > queries on EMR. So for the purpose of experimenting with chukwa we >> > configured chukwa on a 2 node EMR cluster. The collector and the >> > agent are running and also the system logs are generated but the >> > hadoop mapred and rpc /jvm logs are not being generated. Chukwa >> > version : 0.4 and the hadoop version :0.20.205 >> > >> > >> > >> > My hadoop-metrics.properties is as below : >> > >> > >> > # Licensed to the Apache Software Foundation (ASF) under one or more >> > >> > # contributor license agreements. See the NOTICE file distributed with >> > >> > # this work for additional information regarding copyright ownership. >> > >> > # The ASF licenses this file to You under the Apache License, Version >> > 2.0 >> > >> > # (the "License"); you may not use this file except in compliance with >> > >> > # the License. You may obtain a copy of the License at >> > >> > # >> > >> > # http://www.apache.org/licenses/LICENSE-2.0 >> > >> > # >> > >> > # Unless required by applicable law or agreed to in writing, software >> > >> > # distributed under the License is distributed on an "AS IS" BASIS, >> > >> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> > implied. >> > >> > # See the License for the specific language governing permissions and >> > >> > # limitations under the License. >> > >> > >> > >> > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor >> > >> > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost >> > >> > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093 >> > >> > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd >> > >> > log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout >> > >> > log4j.appender.chukwa.rpc.layout.ConversionPattern=%d{ISO8601} %p %c: >> > %m%n >> > >> > log4j.appender.chukwa.rpc.Dir=/tmp/chukwa/log/metrics >> > >> > >> > >> > log4j.appender.chukwa.dfs.recordType=HadoopMetricsProcessor >> > >> > log4j.appender.chukwa.dfs.chukwaClientHostname=localhost >> > >> > log4j.appender.chukwa.dfs.chukwaClientPortNum=9093 >> > >> > log4j.appender.chukwa.dfs.DatePattern=.yyyy-MM-dd >> > >> > log4j.appender.chukwa.dfs.layout=org.apache.log4j.PatternLayout >> > >> > log4j.appender.chukwa.dfs.layout.ConversionPattern=%d{ISO8601} %p %c: >> > %m%n >> > >> > log4j.appender.chukwa.dfs.Dir=/tmp/chukwa/log/metrics >> > >> > >> > >> > log4j.appender.chukwa.mapred.recordType=HadoopMetricsProcessor >> > >> > log4j.appender.chukwa.mapred.chukwaClientHostname=localhost >> > >> > log4j.appender.chukwa.mapred.chukwaClientPortNum=9093 >> > >> > log4j.appender.chukwa.mapred.DatePattern=.yyyy-MM-dd >> > >> > log4j.appender.chukwa.mapred.layout=org.apache.log4j.PatternLayout >> > >> > log4j.appender.chukwa.mapred.layout.ConversionPattern=%d{ISO8601} %p %c: >> > %m%n >> > >> > log4j.appender.chukwa.mapred.Dir=/tmp/chukwa/log/metrics >> > >> > >> > >> > I have also copied chukwa-hadoop-*-client.jar and json.jar to >> > HADOOP_HOME/lib, and restarted hadoop, but still the logs were not >> > generated. >> > >> > Also when I run telnet localhost 9093, the jvm or other hadoop metrics >> > are not running, but the other system adapters are running correctly. >> > >> > Can someone help me here.. >> > >> > Thanks >> > Meena > >
