RE: Hadoop - non disk based sorting?

Hi Bobby, You are right that the Map outputs when copied will be spilled to the disk, but in case the the reducer cannot accomodate the copy inmemory. (shuffleInMemory and shuffleToDisk are chosen by rammanager based on inmemory size) But according to the stack trace provided by Mingxi, >o

Re: Snow Leopard Compilation Help

Ronald, Please take a look at https://issues.apache.org/jira/browse/HADOOP-7147, and https://issues.apache.org/jira/browse/HADOOP-7824 - milind On 12/1/11 5:31 PM, "Ronald Petty" wrote: >Alejandro, > >I suppose I will give it a go since that is the computer I have. I tried >searching on JIRA

Re: Snow Leopard Compilation Help

Alejandro, I suppose I will give it a go since that is the computer I have. I tried searching on JIRA for issues mac related but its hard for me to tell which ones might be related or not. Should I just figure it out and email the list with my fix (if I find one?) Ron On Thu, Dec 1, 2011 at 7:

Re: Snow Leopard Compilation Help

Ron, Hadoop native currently does not compile in Mac OS X. There have been some JIRAs to fix that, but nobody took on them. Thanks. Alejandro On Thu, Dec 1, 2011 at 3:55 PM, Ronald Petty wrote: > Hello, > > I am new to Hadoop development and seem to be stuck on building with Snow > Leopard.

Snow Leopard Compilation Help

Hello, I am new to Hadoop development and seem to be stuck on building with Snow Leopard. Here is what is going on: 1. svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk 2. wget http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz 3. tar ... proto...gz;

Re: how to check which scheduler is currently running on hadoop

Depending on the version of Hadoop you are using, you can goto http://:50030/scheduler to check. This will work from hadoop-0.20.203 onwards. Arun Sent from my iPhone On Nov 30, 2011, at 5:38 PM, shivam tiwari wrote: > Hi, > > please tell me how I can check which scheduler is currently runnin

[jira] [Resolved] (HADOOP-7877) Federation: update Balancer documentation

[ https://issues.apache.org/jira/browse/HADOOP-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HADOOP-7877. Resolution: Fixed Fix Version/s: 0.23.1 0.24.0

Re: Hadoop - non disk based sorting?

I've seen this issue in jobs with many many map tasks and small reducer heaps. There is some heap space needed for the actual map completion events, etc, and that isn't accounted for in determining when to spill the fetch outputs to disk. Would be a nice patch to add code that calculates the in-mem

Re: Hadoop - non disk based sorting?

Mingxi, My understanding was that just like with the maps that when a reducer's in memory buffer fills up it too will spill to disk as part of the sort. In fact I think it uses the exact same code for doing the sort as the map does. There may be an issue where your sort buffer is some how too

Re: how to check which scheduler is currently running on hadoop

On 01/12/11 12:03, shivam tiwari wrote: Is there any other way by which we can check that other scheduler is actually running and not the default one, rather than checking mapred-site.xml file If you are really worried you could kill -QUIT the JT process and look in the stack traces.

Re: how to check which scheduler is currently running on hadoop

Shivam, Visit JobTrackerHost/conf to see the taskScheduler config that's in effect on the runtime. Visit JobTrackerHost/scheduler to see a scheduler web UI if its put one up (default does not provide any, others may). May I ask why you're looking to confirm? Is something not working the way you

Re: how to check which scheduler is currently running on hadoop

Is there any other way by which we can check that other scheduler is actually running and not the default one, rather than checking mapred-site.xml file On Thu, Dec 1, 2011 at 2:10 AM, Praveen Sripati wrote: > Hi, > > Check the mapreduce.jobtracker.taskscheduler property in the > mapred-site.xml,

RE: Hadoop - non disk based sorting?

Hi Mingxi , >So, why when map outputs are huge, reducer will not able to copy them? The Reducer will copy the Map output into its inmemory buffer. When the Reducer JVM doesnt have enough memory to accomodate the Map output, then it leads to OutOfMemoryException. >Can you please kindly explain

RE: Hadoop - non disk based sorting?

Hi Mingxi , >So, why when map outputs are huge, reducer will not able to copy them? The Reducer will copy the Map output into its inmemory buffer. When the Reducer JVM doesnt have enough memory to accomodate the Map output, then it leads to OutOfMemoryException. >Can you please kindly explain

Build failed in Jenkins: Hadoop-Common-0.23-Build #82

See Changes: [mahadev] MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue. (Jonathan Eagles via mahadev) - Merging r1208999 from trunk [mahadev] MAPREDUCE-3463. Second AM fails to recover properly when

[jira] [Created] (HADOOP-7876) Allow access to BlockKey/DelegationKey endoded key for RPC over protobuf

Allow access to BlockKey/DelegationKey endoded key for RPC over protobuf Key: HADOOP-7876 URL: https://issues.apache.org/jira/browse/HADOOP-7876 Project: Hadoop Common

Re: Which branch for my patch?

Thanks, I'll get busy creating a new patch over the next few days. Niels Basjes On Wed, Nov 30, 2011 at 18:51, Eli Collins wrote: > Hey Niels, > > Thanks for contributing. The nest place to contribute new features is > to trunk. It's currently an easy merge from trunk to branch 23 to get > it

[jira] [Created] (HADOOP-7875) Add helper class to unwrap RemoteException from ServiceException thrown on protobuf based RPC

Add helper class to unwrap RemoteException from ServiceException thrown on protobuf based RPC - Key: HADOOP-7875 URL: https://issues.apache.org/jira/browse/HADOOP-7875