Yes, the data is written to HDFS, but you probably need to run demux job to organize the data into the form that is suitable for you.
regards, Eric On Wed, Jan 9, 2013 at 2:54 AM, Shouvanik Haldar <[email protected] > wrote: > I am trying to find out a check point , so that I can say that Chukwa is > running and collecting logs > > > I configured the /chukwa/etc/chukwa/initial_adaptors > with following > > add FileTailer > /home/ec2-user/hadoop-1.0.4/logs/hadoop-ec2-user-datanode-domU-12-31-39-07-19-3B.log > 0 > > . > then I started bin/chukwa agent. > > Started the hadoop daemons. > > then, bin/chukwa collector. > > Next, I check the site > "http://ec2-204-236-192-203.compute-1.amazonaws.com:8080/chukwa?ping=true"; > > with following statistics > > Date:1357728723103 > Now:1357728765604 > numberHTTPConnection in time window:17 > numberchunks in time window:119 > lifetimechunks:23029 > > > and when I check hdfs I see > > -rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 > /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done > -rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14 > /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done > > -rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19 > /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done > -rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24 > /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done > > -rw-r--r-- 1 ec2-user supergroup 386337 2013-01-08 11:36 > /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done > -rw-r--r-- 1 ec2-user supergroup 479656 2013-01-08 11:41 > /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done > > -rw-r--r-- 1 ec2-user supergroup 487748 2013-01-08 11:46 > /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done > -rw-r--r-- 1 ec2-user supergroup 479696 2013-01-08 11:51 > /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done > > -rw-r--r-- 1 ec2-user supergroup 488030 2013-01-08 11:56 > /chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done > -rw-r--r-- 1 ec2-user supergroup 479754 2013-01-08 12:01 > /chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done > > -rw-r--r-- 1 ec2-user supergroup 488067 2013-01-08 12:06 > /chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done > -rw-r--r-- 1 ec2-user supergroup 479961 2013-01-08 12:11 > /chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done > > -rw-r--r-- 1 ec2-user supergroup 487976 2013-01-08 12:16 > /chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done > -rw-r--r-- 1 ec2-user supergroup 479863 2013-01-08 12:21 > /chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done > > -rw-r--r-- 1 ec2-user supergroup 649168 2013-01-08 12:42 > /chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done > -rw-r--r-- 1 ec2-user supergroup 483972 2013-01-08 12:47 > /chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done > > -rw-r--r-- 1 ec2-user supergroup 483417 2013-01-08 12:52 > /chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done > -rw-r--r-- 1 ec2-user supergroup 576711 2013-01-08 13:06 > /chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done > > -rw-r--r-- 1 ec2-user supergroup 483877 2013-01-08 13:11 > /chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done > -rw-r--r-- 1 ec2-user supergroup 484352 2013-01-08 13:16 > /chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done > > -rw-r--r-- 1 ec2-user supergroup 486158 2013-01-08 13:21 > /chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done > -rw-r--r-- 1 ec2-user supergroup 483925 2013-01-08 13:26 > /chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done > > -rw-r--r-- 1 ec2-user supergroup 629788 2013-01-08 13:37 > /chukwa/logs/201308133253308_domU12313907193B_7382697b13c1a5d14db8000.done > -rw-r--r-- 1 ec2-user supergroup 485716 2013-01-08 13:42 > /chukwa/logs/201308133753394_domU12313907193B_7382697b13c1a5d14db7ffb.done > > -rw-r--r-- 1 ec2-user supergroup 486475 2013-01-08 13:47 > /chukwa/logs/201308134253394_domU12313907193B_7382697b13c1a5d14db7ff5.done > -rw-r--r-- 1 ec2-user supergroup 481330 2013-01-08 13:52 > /chukwa/logs/201308134753394_domU12313907193B_7382697b13c1a5d14db7fef.done > > -rw-r--r-- 1 ec2-user supergroup 488138 2013-01-08 13:57 > /chukwa/logs/201308135253394_domU12313907193B_7382697b13c1a5d14db7fe9.done > -rw-r--r-- 1 ec2-user supergroup 484294 2013-01-08 14:02 > /chukwa/logs/201308135753394_domU12313907193B_7382697b13c1a5d14db7fe3.done > > -rw-r--r-- 1 ec2-user supergroup 480689 2013-01-08 14:07 > /chukwa/logs/201308140253394_domU12313907193B_7382697b13c1a5d14db7fdd.done > -rw-r--r-- 1 ec2-user supergroup 488018 2013-01-08 14:12 > /chukwa/logs/201308140753395_domU12313907193B_7382697b13c1a5d14db7fd7.done > > -rw-r--r-- 1 ec2-user supergroup 287583 2013-01-08 14:23 > /chukwa/logs/201308141253395_domU12313907193B_7382697b13c1a5d14db7fd1.done > -rw-r--r-- 1 ec2-user supergroup 527781 2013-01-08 14:27 > /chukwa/logs/201308142254785_domU12313907193B_3065fa8513c1a8ae1598000.done > > -rw-r--r-- 1 ec2-user supergroup 483844 2013-01-08 14:32 > /chukwa/logs/201308142754897_domU12313907193B_3065fa8513c1a8ae1597ff9.done > -rw-r--r-- 1 ec2-user supergroup 483447 2013-01-08 14:37 > /chukwa/logs/201308143254897_domU12313907193B_3065fa8513c1a8ae1597ff3.done > > -rw-r--r-- 1 ec2-user supergroup 483915 2013-01-08 14:42 > /chukwa/logs/201308143754897_domU12313907193B_3065fa8513c1a8ae1597fed.done > -rw-r--r-- 1 ec2-user supergroup 483902 2013-01-08 14:47 > /chukwa/logs/201308144254897_domU12313907193B_3065fa8513c1a8ae1597fe7.done > > -rw-r--r-- 1 ec2-user supergroup 484187 2013-01-08 14:52 > /chukwa/logs/201308144754897_domU12313907193B_3065fa8513c1a8ae1597fe1.done > -rw-r--r-- 1 ec2-user supergroup 295291 2013-01-08 17:04 > /chukwa/logs/201308145254897_domU12313907193B_3065fa8513c1a8ae1597fdb.done > > -rw-r--r-- 1 ec2-user supergroup 524107 2013-01-08 17:08 > /chukwa/logs/201308170310783_domU123139078691_50774b0013c1b1d9bd48000.done > -rw-r--r-- 1 ec2-user supergroup 493338 2013-01-08 17:13 > /chukwa/logs/201308170810879_domU123139078691_50774b0013c1b1d9bd47ff7.done > > -rw-r--r-- 1 ec2-user supergroup 485752 2013-01-08 17:18 > /chukwa/logs/201308171310880_domU123139078691_50774b0013c1b1d9bd47ff1.done > -rw-r--r-- 1 ec2-user supergroup 566482 2013-01-08 17:26 > /chukwa/logs/201308172154776_domU123139078691_2f8337f413c1b2ec27b8000.done > > -rw-r--r-- 1 ec2-user supergroup 483669 2013-01-08 17:31 > /chukwa/logs/201308172654963_domU123139078691_2f8337f413c1b2ec27b7ffa.done > -rw-r--r-- 1 ec2-user supergroup 483893 2013-01-08 17:36 > /chukwa/logs/201308173154963_domU123139078691_2f8337f413c1b2ec27b7ff3.done > > -rw-r--r-- 1 ec2-user supergroup 483996 2013-01-08 17:41 > /chukwa/logs/201308173654964_domU123139078691_2f8337f413c1b2ec27b7fec.done > -rw-r--r-- 1 ec2-user supergroup 483918 2013-01-08 17:46 > /chukwa/logs/201308174154964_domU123139078691_2f8337f413c1b2ec27b7fe5.done > > -rw-r--r-- 1 ec2-user supergroup 484337 2013-01-08 17:51 > /chukwa/logs/201308174654964_domU123139078691_2f8337f413c1b2ec27b7fde.done > -rw-r--r-- 1 ec2-user supergroup 103077 2013-01-09 07:08 > /chukwa/logs/201308175154964_domU123139078691_2f8337f413c1b2ec27b7fd7.done > > -rw-r--r-- 1 ec2-user supergroup 539279 2013-01-09 07:13 > /chukwa/logs/201309070837366_ip10461114_5f07286f13c1e23a2908000.done > -rw-r--r-- 1 ec2-user supergroup 450569 2013-01-09 07:18 > /chukwa/logs/201309071337486_ip10461114_5f07286f13c1e23a2907ff9.done > > -rw-r--r-- 1 ec2-user supergroup 450802 2013-01-09 07:23 > /chukwa/logs/201309071837486_ip10461114_5f07286f13c1e23a2907ff2.done > -rw-r--r-- 1 ec2-user supergroup 450749 2013-01-09 07:28 > /chukwa/logs/201309072337486_ip10461114_5f07286f13c1e23a2907feb.done > > -rw-r--r-- 1 ec2-user supergroup 545135 2013-01-09 08:19 > /chukwa/logs/201309081415268_ip10461114_b00255013c1e5fb8f78000.done > -rw-r--r-- 1 ec2-user supergroup 457343 2013-01-09 08:24 > /chukwa/logs/201309081915384_ip10461114_b00255013c1e5fb8f77ff8.done > > -rw-r--r-- 1 ec2-user supergroup 452951 2013-01-09 08:29 > /chukwa/logs/201309082415384_ip10461114_b00255013c1e5fb8f77ff0.done > -rw-r--r-- 1 ec2-user supergroup 534336 2013-01-09 08:40 > /chukwa/logs/201309083503153_ip10461114_4c587e7413c1e72c3ba8000.done > > -rw-r--r-- 1 ec2-user supergroup 454511 2013-01-09 08:45 > /chukwa/logs/201309084003370_ip10461114_4c587e7413c1e72c3ba7ff8.done > -rw-r--r-- 1 ec2-user supergroup 451026 2013-01-09 08:50 > /chukwa/logs/201309084503370_ip10461114_4c587e7413c1e72c3ba7fef.done > > -rw-r--r-- 1 ec2-user supergroup 447653 2013-01-09 08:55 > /chukwa/logs/201309085003370_ip10461114_4c587e7413c1e72c3ba7fe6.done > -rw-r--r-- 1 ec2-user supergroup 451091 2013-01-09 09:00 > /chukwa/logs/201309085503370_ip10461114_4c587e7413c1e72c3ba7fdd.done > > -rw-r--r-- 1 ec2-user supergroup 449429 2013-01-09 09:05 > /chukwa/logs/201309090003371_ip10461114_4c587e7413c1e72c3ba7fd4.done > -rw-r--r-- 1 ec2-user supergroup 456627 2013-01-09 09:10 > /chukwa/logs/201309090503371_ip10461114_4c587e7413c1e72c3ba7fcb.done > > -rw-r--r-- 1 ec2-user supergroup 451078 2013-01-09 09:15 > /chukwa/logs/201309091003371_ip10461114_4c587e7413c1e72c3ba7fc2.done > -rw-r--r-- 1 ec2-user supergroup 447648 2013-01-09 09:20 > /chukwa/logs/201309091503371_ip10461114_4c587e7413c1e72c3ba7fb9.done > > -rw-r--r-- 1 ec2-user supergroup 451155 2013-01-09 09:25 > /chukwa/logs/201309092003371_ip10461114_4c587e7413c1e72c3ba7fb0.done > -rw-r--r-- 1 ec2-user supergroup 455010 2013-01-09 09:30 > /chukwa/logs/201309092503371_ip10461114_4c587e7413c1e72c3ba7fa7.done > > -rw-r--r-- 1 ec2-user supergroup 451450 2013-01-09 09:35 > /chukwa/logs/201309093003372_ip10461114_4c587e7413c1e72c3ba7f9e.done > -rw-r--r-- 1 ec2-user supergroup 447741 2013-01-09 09:40 > /chukwa/logs/201309093503372_ip10461114_4c587e7413c1e72c3ba7f95.done > > -rw-r--r-- 1 ec2-user supergroup 451407 2013-01-09 09:45 > /chukwa/logs/201309094003372_ip10461114_4c587e7413c1e72c3ba7f8c.done > -rw-r--r-- 1 ec2-user supergroup 451304 2013-01-09 09:50 > /chukwa/logs/201309094503373_ip10461114_4c587e7413c1e72c3ba7f83.done > > -rw-r--r-- 1 ec2-user supergroup 454591 2013-01-09 09:55 > /chukwa/logs/201309095003374_ip10461114_4c587e7413c1e72c3ba7f7a.done > -rw-r--r-- 1 ec2-user supergroup 451044 2013-01-09 10:00 > /chukwa/logs/201309095503374_ip10461114_4c587e7413c1e72c3ba7f71.done > > -rw-r--r-- 1 ec2-user supergroup 448113 2013-01-09 10:05 > /chukwa/logs/201309100003374_ip10461114_4c587e7413c1e72c3ba7f68.done > -rw-r--r-- 1 ec2-user supergroup 451071 2013-01-09 10:10 > /chukwa/logs/201309100503374_ip10461114_4c587e7413c1e72c3ba7f5f.done > > -rw-r--r-- 1 ec2-user supergroup 451235 2013-01-09 10:15 > /chukwa/logs/201309101003374_ip10461114_4c587e7413c1e72c3ba7f56.done > -rw-r--r-- 1 ec2-user supergroup 454562 2013-01-09 10:20 > /chukwa/logs/201309101503374_ip10461114_4c587e7413c1e72c3ba7f4d.done > > -rw-r--r-- 1 ec2-user supergroup 451237 2013-01-09 10:25 > /chukwa/logs/201309102003375_ip10461114_4c587e7413c1e72c3ba7f44.done > -rw-r--r-- 1 ec2-user supergroup 451241 2013-01-09 10:30 > /chukwa/logs/201309102503375_ip10461114_4c587e7413c1e72c3ba7f3b.done > > -rw-r--r-- 1 ec2-user supergroup 447653 2013-01-09 10:35 > /chukwa/logs/201309103003375_ip10461114_4c587e7413c1e72c3ba7f32.done > -rw-r--r-- 1 ec2-user supergroup 451351 2013-01-09 10:40 > /chukwa/logs/201309103503375_ip10461114_4c587e7413c1e72c3ba7f29.done > > -rw-r--r-- 1 ec2-user supergroup 451706 2013-01-09 10:45 > /chukwa/logs/201309104003376_ip10461114_4c587e7413c1e72c3ba7f20.done > -rw-r--r-- 1 ec2-user supergroup 450777 2013-01-09 10:50 > /chukwa/logs/201309104503375_ip10461114_4c587e7413c1e72c3ba7f17.done > > > Does it mean, Chukwa collected the logs. > > Regards, > Shouvanik > > > -- > Thanks, > *Shouvanik* >
