State of Art in Hadoop Log aggregation

2013-10-10 Thread Sagar Mehta
Hi Guys, We have fairly decent sized Hadoop cluster of about 200 nodes and was wondering what is the state of art if I want to aggregate and visualize Hadoop ecosystem logs, particularly 1. Tasktracker logs 2. Datanode logs 3. Hbase RegionServer logs One way is to use something like a F

Re: Automatically mapping a job submitted by a particular user to a specific hadoop map-reduce queue

2013-04-26 Thread Sagar Mehta
to. > > Thanks, > +Vinod Kumar Vavilapalli > Hortonworks Inc. > http://hortonworks.com/ > > On Apr 24, 2013, at 6:22 PM, Sagar Mehta wrote: > > Hi Guys, > > We have a general purpose Hive cluster [about 200 nodes] which is used for > various jobs like > >

Re: Automatically mapping a job submitted by a particular user to a specific hadoop map-reduce queue

2013-04-26 Thread Sagar Mehta
the fair scheduler (or other >> schedulers, as far as I know), but a JIRA has been filed recently that >> addresses a similar need. Would >> https://issues.apache.org/jira/browse/MAPREDUCE-5132 work for what >> you're trying to do? If not, would you mind filing a n

Re: Automatically mapping a job submitted by a particular user to a specific hadoop map-reduce queue

2013-04-26 Thread Sagar Mehta
gt; > -Sandy > > > On Wed, Apr 24, 2013 at 6:22 PM, Sagar Mehta wrote: > >> Hi Guys, >> >> We have a general purpose Hive cluster [about 200 nodes] which is used >> for various jobs like >> >>- Production >>- Experimental/Research &g

Automatically mapping a job submitted by a particular user to a specific hadoop map-reduce queue

2013-04-24 Thread Sagar Mehta
Hi Guys, We have a general purpose Hive cluster [about 200 nodes] which is used for various jobs like - Production - Experimental/Research - Adhoc queries We are using the fair-share scheduler to schedule them and for this we have corresponding 3 pools in the scheduler. *Here is what w