Hadoop-on-demand and torque

If I understand it right HOD is mentioned mainly for merging existing HPC clusters with hadoop and for testing purposes.. I cannot find what is the role of Torque here (just initial nodes allocation?) and which is the default scheduler of HOD ? Probably the scheduler from the hadoop distribution?

Re: Distributing MapReduce on a computer cluster

For distribution of load you can start reading some chapters from different types of hadoop scheduler. I have not yet studied other implementation like hadoop, however a very simplified version of distribution concept is the following: a) Tasktracker ask for work (heartbeat consist of a status of

Re: Algorithms used in fairscheduler 0.20.205

Anyone? On 19 April 2012 17:34, Merto Mertek wrote: > I could find that the closest doc matching the current implementation of > the fairscheduler could be find in this > document<http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-55.html>from > Matei Zaharia et al..

Algorithms used in fairscheduler 0.20.205

I could find that the closest doc matching the current implementation of the fairscheduler could be find in this documentfrom Matei Zaharia et al.. Another documented from delay scheduling can be found from year 2010.. a) I am inte

Re: Fairscheduler - disable "default" pool

? > For that, "mapred.fairscheduler.allow.undeclared.pools" needs to be > set to false. > > On Wed, Mar 14, 2012 at 1:36 AM, Merto Mertek wrote: > > Thanks for your workaround, but I think that with this you just put a > > constraint on the pool that it will not a

Re: Fairscheduler - disable "default" pool

: > We do it here by setting this: > > 0 > > So that you _must_ have a pool (that's configured with a different > maxRunningJobs) in order to run jobs. > > Hope this helps, > > J-D > > On Tue, Mar 13, 2012 at 10:49 AM, Merto Mertek > wrote: > > I know

Fairscheduler - disable "default" pool

I know that by design all unmarked jobs goes to that pool, however I am doing some testing and I am interested if is possible to disable it.. Thanks

Re: Hadoop fair scheduler doubt: allocate jobs to pool

hen I did ${user.name} it worked! and created a pool with the user > name. > > > > On Thu, Mar 1, 2012 at 8:03 PM, Merto Mertek wrote: > > > From the fairscheduler docs I assume the following should work: > > > > > > mapred.fairscheduler.poolnameproperty

Re: Hadoop fair scheduler doubt: allocate jobs to pool

>From the fairscheduler docs I assume the following should work: mapred.fairscheduler.poolnameproperty pool.name pool.name ${mapreduce.job.group.name} which means that the default pool will be the group of the user that has submitted the job. In your case I think that allocations

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

pshot_imax_time.rrd): > > > > > > > > > /var/lib/ganglia/rrds/hadoop/slave1/metricssystem.MetricsSystem.snapshot_imax_time.rrd: > > > converting '4.9E-324' to float: Numerical result out of range > > > Feb 15 09:39:14 master gmetad[16487]: RRD_upda

Re: Tasktracker fails

stackoverflow.com/questions/9400739/hadoop-globstatus-and-deflate-files > > On Wed, Feb 22, 2012 at 7:39 AM, Merto Mertek wrote: > > > Hm.. I would try first to stop all the deamons wtih > > $haddop_home/bin/stop-all.sh. Afterwards check that on the master and one > > of the sl

Re: Tasktracker fails

Hm.. I would try first to stop all the deamons wtih $haddop_home/bin/stop-all.sh. Afterwards check that on the master and one of the slaves no deamons are running (jps). Maybe you could try to check if your conf on tasktrackers for the jobtracker is pointing to the right place (mapred-site.xml). Do

Re: Dynamic changing of slaves

I think that job configuration does not allow you such setup, however maybe I missed something.. Probably I would tackle this problem from the scheduler source. The default one is JobQueueTaskScheduler which preserves a fifo based queue. When a tasktracker (your slave) tells the jobtracker that i

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

; > couple metrics emitted TO gmetad (FROM all the nodes in the cluster), so > > without copying it over to every node in the cluster, gmetad will still > > likely receive some bad metrics. > > > > Varun > > > > > > On Wed, Feb 8, 2012 at 6:19 PM, Merto

Compile and deploy a new version of hadoop

I am having some troubles in understanding how the whole stuff works.. Compiling with ant works ok and I am able to compile a jar which is afterwards deployed to the cluster. On the cluster I've set the HADOOP_CLASSPATH variable to point just to jar files in the lib folder ($HD_HOME/lib/*.jar), wh

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

from my Dropbox account): > > - http://dl.dropbox.com/u/4366344/gmetadBufferOverflow.Hadoop.patch > - http://dl.dropbox.com/u/4366344/gmetadBufferOverflow.gmetad.patch > > Here's hoping this works for you, > > Varun > On Tue, Feb 7, 2012 at 6:00 PM, Merto Mertek wrote: &

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

t the hadoop metrics enabled everything is smooth. >> Regards >> >> Mete >> >> On Tue, Feb 7, 2012 at 4:58 AM, Merto Mertek wrote: >> >> > I have tried to run it but it repeats crashing.. >> > >> > - When you start gmetad and Hadoop i

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

ave checked and all of them are versioned 3.2.0. Hope it helps.. > > Bear in mind that this only addresses the gmetad coring issue - the > warnings emitted about '4.9E-324' being out of range will continue, but I > know what's causing that as well (and hope tha

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

ch > > available for this, HADOOP-4675. As of November 2010, this patch has been > > rolled into the mainline for 0.20.2 and later. To use the Ganglia 3.1 > > protocol in place of the 3.0, substitute > > org.apache.hadoop.metrics.ganglia.GangliaContext31 for > > org.apache.hadoop.metrics.ganglia.Gang

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

olution? Would you rather suggest to use ganglia 3.0.x with the old protocol and leave the version >3.1 for further releases? any help is realy appreciated... On 1 February 2012 04:04, Merto Mertek wrote: > I would be glad to hear that too.. I've setup the following: > > Hadoop

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

I would be glad to hear that too.. I've setup the following: Hadoop 0.20.205 Ganglia Front 3.1.7 Ganglia Back *(gmetad)* 3.1.7 RRDTool 1.4.5. -> i had some troubles installing 1.4.4 Ganglia works just in case hadoop is not running, so metrics are not publshed to gmetad

Similar frameworks like hadoop and taxonomy of distributed computing

Hi, I was wondering if anyone knows any paper discussing and comparing the mentioned topic. I am a little bit confused about the classification of hadoop.. Is it a /cluster/comp grid/ a mix of them? What is hadoop in relation with a cloud - probably just a technology that enables cloud services..

Re: Desperate!!!! Expanding,shrinking cluster or replacing failed nodes.

I followed the same tutorial as you. If I am not wrong the problem arise because you first tried to run a node as single node and then joining it to the cluster (like Arpit mentioned). After testing that the new node works ok try to delete content in directory /app/hadoop/tmp/ and insert a new node

Re: Configure hadoop scheduler

duler to FairScheduler. > > Sent from my iPhone > > On Dec 20, 2011, at 8:51 AM, Merto Mertek wrote: > > > Hi, > > > > I am having problems with changing the default hadoop scheduler (i assume > > that the default scheduler is a FIFO scheduler). > >

Configure hadoop scheduler

Hi, I am having problems with changing the default hadoop scheduler (i assume that the default scheduler is a FIFO scheduler). I am following the guide located in hadoop/docs directory however I am not able to run it. Link for scheduling administration returns an http error 404 ( http://localhos

Re: TestFairScheduler failing - version 0.20. security 204

scheduler build/jar to a cluster - try it on a working cluster Is there any other option how to try a new functionality locally or in any other way? Any comments and suggestion are welcomed Thank you.. On 17 December 2011 21:58, Merto Mertek wrote: > Hi, > > I am having some proble

TestFairScheduler failing - version 0.20. security 204

Hi, I am having some problems with running the following test file org.apache.hadoop.mapred.TestFairScheduler Nearly all test fails, most of them with the error: javalang.runtimeexception: COULD NOT START JT. Here is a trace . Code was checkout from the svn branch,

Re: Environment consideration for a research on scheduling

both the bin installs as well as the user >> creation. >> >> Your VM route will most likely work but I can imagine the amount of >> hiccups during migration from that to the real cluster will not make it >> worth your time. >> >> Matt >> >> -Orig

Re: Environment consideration for a research on scheduling

t to the real cluster will not make it worth your > time. > > Matt > > -Original Message- > From: Merto Mertek [mailto:masmer...@gmail.com] > Sent: Friday, September 23, 2011 10:00 AM > To: common-user@hadoop.apache.org > Subject: Environment consideration for a re

Environment consideration for a research on scheduling

Hi, in the first phase we are planning to establish a small cluster with few commodity computer (each 1GB, 200GB,..). Cluster would run ubuntu server 10.10 and a hadoop build from the branch 0.20.204 (i had some issues with version 0.20.203 with missing libraries

Re: Unsubscribe from jira issues

hehe :) you are right :) On 23 September 2011 16:21, Harsh J wrote: > Merto, > > Am sure your mail client has some form of filtering available in that case! > :-) > > On Fri, Sep 23, 2011 at 7:49 PM, Merto Mertek wrote: > > Probably there is not any option just to disa

Re: Unsubscribe from jira issues

opened/resolved/reopened > messages. The common-issues receives everything. > > On Fri, Sep 23, 2011 at 7:27 PM, Merto Mertek wrote: > > Hi, > > i am receiving messages from two mailing lists > ("common-dev","common-user") > > and I would like to

Unsubscribe from jira issues

Hi, i am receiving messages from two mailing lists ("common-dev","common-user") and I would like to disable receiving msg from jira. I am not a member of "common-issues-unsubscribe" list. Can I anyhow disable this? Thank you