Data should store in hadoop_jvm_*_week, hadoop_mapred_*_week, hadoop_rpc_*_week. It looks like the system is working properly. There should be some data on HDFS in /chukwa/repos that has some data for each data type. Check if files exists in there.
regards, Eric 2011/7/2 孙长 <[email protected]>: > > my hadoop version is 0.20.1 > i telnet localhost 9093 and list, then shows: > adaptor_8567b8b00a5dc746ccd5b8d179873db1) > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60 > /usr/bin/top -b -n 1 -c 2110354484 > adaptor_4be0d8af3bc253eef29983e5dbd4f0d5) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309528269962.log > 150865 > adaptor_b5382452cfdc2ae81bd45d8126804843) > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60 > /usr/bin/iostat -x -k 55 2 4771609 > adaptor_f0b4a4e9d4549f38f633c451e06e290f) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > HadoopLog 0 > /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-jobtracker-hadoop10-131-4-70.log > 63614063 > adaptor_8890f1df2c09c514450025c0502f473d) > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60 > /usr/bin/sar -q -r -n ALL 55 129031858 > adaptor_bbdbf9f9ccd112b1d2973aabf4e7c422) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578850828.log > 18561 > adaptor_5df3de68e136f337c5a46b9bf04637d1) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > HadoopLog 0 > /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-secondarynamenode-hadoop10-131-4-70.log > 51747 > adaptor_31842d1822977c3c492c725cc8b81bda) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527999599.log > 380835 > adaptor_0e10bfd94287c9a1c2983c7e103e39f0) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527883608.log > 1060785 > adaptor_46d675cf8ccd92e245509b19f28dcea8) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309527883619.log > 952265 > adaptor_1cd9e4134b790becae04026c2fda49ff) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582186581.log > 76459 > adaptor_5a8f9af70b0e591b034f672f4340371c) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582370136.log > 74779 > adaptor_fa36c905bcb3701ae4183185ebb8af01) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527883616.log > 380397 > adaptor_a20e4ffdb7d3fad38732f13fc5aa54a5) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309527918692.log > 801761 > adaptor_72174dc3ec7b8f3871c5d277a0f45125) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582273055.log > 143061 > adaptor_e24367b38aa7970ea475f67a850fd019) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > chukwaAgent 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaAgent.log 366838 > adaptor_1ec2f74ed77d549837b7d5d33db64c7e) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > mapred 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309527999602.log > 10253607 > adaptor_b74c14fdd101db4a7c9bd7190c230428) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309578883752.log > 18568 > adaptor_16b8b916c39b571a2d38c7ec5bda0a62) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578884932.log > 31245 > adaptor_4b887b4d345650292baa10290cd34693) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > mapred 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-mapred-1309582370140.log > 2216270 > adaptor_a2e4104972a498bee2619f6ee0b48efa) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > chukwaHttpSender 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chukwaHttpSender.log > 449552 > adaptor_9532d27cd32d9c64757583ea28b37664) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > HadoopLog 0 > /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/hadoop-distribute-hadoop-boss-namenode-hadoop10-131-4-70.log > 353185023 > adaptor_12d05a003a847c3fc1b2c850decbc5bd) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309578850821.log > 46459 > adaptor_d096625c34f55d7d9c5bc9950aa51f04) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309500469601.log > 242968 > adaptor_ea4c5487a8a14a91da47640303f58eb7) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309578850831.log > 45630 > adaptor_52e2b02867360ff8b60da5fc4acbcf6e) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582186584.log > 190797 > adaptor_17e122f11752d4f85bb3771962f4ed0b) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > chunkQueue 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-chunkQueue.log 376527 > adaptor_6be04642c8e5f3bf89bb12333baaa678) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309527917408.log > 382148 > adaptor_136ad19fb21c0d398072457a231baba2) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > jvm 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-jvm-1309582271839.log > 75547 > adaptor_b6ac14ec507719e8e01a5e9af8cc0cfc) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309582633863.log > 28576 > adaptor_6dd77c53faf6f9224a4ffdf3cb77796b) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309582186572.log > 202433 > adaptor_ffdb380f632a5efb0aa3532df247502f) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > dfs 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-dfs-1309500469604.log > 607138 > adaptor_e69999b07d7023e6ba08060c85bd9ad7) > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df > -l 1437897063 > adaptor_ddad8a9e25eb3e3502d7b4d399207ea8) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > HadoopLog 0 > /usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.20.1/bin/../logs/audit.log > 172088797 > adaptor_469551943c09f2cda60cbe3965c35614) > org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLineEscaped > rpc 0 > /tmp/chukwa/log/metrics/chukwa-distribute-hadoop-boss-rpc-1309500469592.log > 797132 > > you see, the adaptors are running correctly, so ,which mysql tables will > save mapred data? > > At 2011-07-02 13:04:33，"Eric Yang" <[email protected]> wrote: > >>Chukwa 0.4 only works with hadoop 0.20.x. If you are using hadoop >>0.2, then it probably doesn't work. >> >>On the nodes that you see the log files, try: >> >>telnet localhost 9093 >>list >> >>Check if the jvm and mapred files are being streamed over chukwa agent. >> >>If it is not, you might need to restart the mapreduce cluster. >> >>regards, >>Eric >> >>2011/7/1 孙长 <[email protected]>: >>> yes, there are rpc and mapred logs in log file >>> >>> hadoop-metrics.properties is like this, and hadoop version is 0.2 >>> >>> # The ASF licenses this file to You under the Apache License, Version 2.0 >>> # (the "License"); you may not use this file except in compliance with >>> # the License. You may obtain a copy of the License at >>> # >>> # http://www.apache.org/licenses/LICENSE-2.0 >>> # >>> # Unless required by applicable law or agreed to in writing, software >>> # distributed under the License is distributed on an "AS IS" BASIS, >>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >>> # See the License for the specific language governing permissions and >>> # limitations under the License. >>> chukwaAgent.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> chukwaAgent.period=60 >>> chukwaAgent.directory=/tmp/chukwa/log/metrics/ >>> >>> chukwaHttpSender.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> chukwaHttpSender.period=60 >>> chukwaHttpSender.directory=/tmp/chukwa/log/metrics/ >>> chunkQueue.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> chunkQueue.period=60 >>> chunkQueue.directory=/tmp/chukwa/log/metrics/ >>> dfs.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> dfs.period=60 >>> dfs.directory=/tmp/chukwa/log/metrics/ >>> dfs.uuid=true >>> jvm.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> jvm.period=60 >>> jvm.directory=/tmp/chukwa/log/metrics/ >>> jvm.uuid=true >>> mapred.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> mapred.period=60 >>> mapred.directory=/tmp/chukwa/log/metrics/ >>> mapred.uuid=true >>> rpc.class=org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext >>> rpc.period=60 >>> rpc.directory=/tmp/chukwa/log/metrics/ >>> rpc.uuid=true >>> >>> At 2011-07-02 11:39:32，"Eric Yang" <[email protected]> wrote: >>> >>>>Check the log directory and see if you see any log file with rpc or >>>>mapred in the file name? >>>>What does the hadoop-metrics.properties look like? What version of >>>>hadoop are you running? >>>> >>>>regards, >>>>Eric >>>> >>>>2011/7/1 孙长 <[email protected]>: >>>>> i know where the problem is , i didn't put log4j and hadoop-metrics file >>>>> into all nodes' hadoop conf dir. >>>>> Now the problem disapears >>>>> but for hadoop metrics, i only got dfs data in mysql, no mapreduce, jvm, rpc >>>>> data, why? >>>>> thank you~ >>>>> >>>>> >>>>> At 2011-07-01 22:59:41，"Eric Yang" <[email protected]> wrote: >>>>> >>>>>>Looks like mapreduce can not find json.jar in its class path. Which >>>>>>version of Hadoop are you using? >>>>>> >>>>>>regards, >>>>>>Eric >>>>>> >>>>>>2011/7/1 孙长 <[email protected]>: >>>>>>> hi,i have copied chukwa-hadoop-*-client.jar and json.jar to >>>>>>> HADOOP_HOME/lib, and restarted hadoop, but the MR job lanched by demux runs >>>>>>> error: >>>>>>> task error is as follows: >>>>>>> >>>>>>> Error: java.lang.ClassNotFoundException: org.json.JSONException >>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:252) >>>>>>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) >>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>> at java.lang.Class.forName(Class.java:169) >>>>>>> at >>>>>>> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.MapProcessorFactory.getProcessor(MapProcessorFactory.java:40) >>>>>>> at >>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:95) >>>>>>> at >>>>>>> org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:64) >>>>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) >>>>>>> >>>>>>>at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) >>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) >>>>>>> at org.apache.hadoop.mapred.Child.main(Child.java:170) >>>>>>> >>>>>>> At 2011-07-01 13:17:42，"Eric Yang" <[email protected]> wrote: >>>>>>> >>>>>>>>Did you copy chukwa-hadoop-*-client.jar and json.jar to >>>>>>>>HADOOP_HOME/lib, and restart hadoop? >>>>>>>> >>>>>>>>regards, >>>>>>>>Eric >>>>>>>> >>>>>>>>2011/6/30 孙长 <[email protected]>: >>>>>>>>> Hi, I am using chukwa 0.4 , and i start a collector and a agent on two >>>>>>>>> nodes, but i only got system metrics in mysql, no hadoop metrics like jvm, >>>>>>>>> dfs, mapred >>>>>>>>> dbAdmin.sh is running background, log4j.properties >>>>>>>>> and hadoop-metrics.properties in hadoop conf dir are set according to user >>>>>>>>> guide >>>>>>>>> no dfs, mapred logs in chukwa/log/metrics dir. >>>>>>>>> thank you ~ >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>> >>> >>> > > >
