Hi, http://flume.apache.org
- Alex On Oct 11, 2013, at 7:36 AM, Sagar Mehta <sagarme...@gmail.com> wrote: > Hi Guys, > > We have fairly decent sized Hadoop cluster of about 200 nodes and was > wondering what is the state of art if I want to aggregate and visualize > Hadoop ecosystem logs, particularly > Tasktracker logs > Datanode logs > Hbase RegionServer logs > One way is to use something like a Flume on each node to aggregate the logs > and then use something like Kibana - > http://www.elasticsearch.org/overview/kibana/ to visualize the logs and make > them searchable. > > However I don't want to write another ETL for the hadoop/hbase logs > themselves. We currently log in to each machine individually to 'tail -F > logs' when there is an hadoop problem on a particular node. > > We want a better way to look at the hadoop logs themselves in a centralized > way when there is an issue without having to login to 100 different machines > and was wondering what is the state of are in this regard. > > Suggestions/Pointers are very welcome!! > > Sagar -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
