Re: State of Art in Hadoop Log aggregation

2013-10-11 Thread Sandy Ryza
Just a clarification: Cloudera Manager is now free for any number of nodes. Ref: http://www.cloudera.com/content/cloudera/en/products/cloudera-manager.html -Sandy On Fri, Oct 11, 2013 at 7:05 AM, DSuiter RDX wrote: > Sagar, > > It sounds like you want a management console. We are using Clouder

Re: State of Art in Hadoop Log aggregation

2013-10-11 Thread DSuiter RDX
Sagar, It sounds like you want a management console. We are using Cloudera Manager, but for 200 nodes you would need to license it, it is only free up to 50 nodes. The FOSS version of this is Ambari, iirc. http://incubator.apache.org/ambari/ Flume will provide a Hadoop-integrated pipeline for in

Re: State of Art in Hadoop Log aggregation

2013-10-11 Thread Alexander Alten-Lorenz
Hi, http://flume.apache.org - Alex On Oct 11, 2013, at 7:36 AM, Sagar Mehta wrote: > Hi Guys, > > We have fairly decent sized Hadoop cluster of about 200 nodes and was > wondering what is the state of art if I want to aggregate and visualize > Hadoop ecosystem logs, particularly > Tasktrack

RE: State of Art in Hadoop Log aggregation

2013-10-11 Thread Smith, Joshua D.
I've used Splunk in the past for log aggregation. It's commercial/proprietary, but I think there's a free version. http://www.splunk.com/ From: Raymond Tay [mailto:raymondtay1...@gmail.com] Sent: Friday, October 11, 2013 1:39 AM To: user@hadoop.apache.org Subject: Re: State of Ar

Re: State of Art in Hadoop Log aggregation

2013-10-10 Thread Pradeep Gollakota
There are plenty of log aggregation tools both open source and commercial off the shelf. Here's some http://devopsangle.com/2012/04/19/8-splunk-alternatives/ My personal recommendation is LogStash. On Thu, Oct 10, 2013 at 10:38 PM, Raymond Tay wrote: > You can try Chukwa which is part of the in

Re: State of Art in Hadoop Log aggregation

2013-10-10 Thread Raymond Tay
You can try Chukwa which is part of the incubating projects under Apache. Tried it before and liked it for aggregating logs. On 11 Oct, 2013, at 1:36 PM, Sagar Mehta wrote: > Hi Guys, > > We have fairly decent sized Hadoop cluster of about 200 nodes and was > wondering what is the state of a