Capacity scheduler and map/reduce jobs

f Queue kick in? Thanks, Aniket Ray

Shuffle tasks getting killed

Hi, I continuously run a series of batch job using Hadoop Map Reduce. I also have a managing daemon that moves data around on the hdfs making way for more jobs to be run. I use capacity scheduler to schedule many jobs in parallel. I see an issue on the Hadoop web monitoring UI at port 50030 which

Re: Shuffle tasks getting killed

n Thu, Sep 23, 2010 at 5:14 PM, cliff palmer wrote: > Aniket, I wonder if these tasks were run as Speculative Execution. Have > you > been able to determine whether the job runs successfully? > HTH > Cliff > > On Thu, Sep 23, 2010 at 12:52 AM, aniket ray wrote: > > &g

Re: Can distcp handle the copy from a non-HDFS cluster to a HDFS cluster?

Hi, Did you get any way to copy from a non-HDFS linux cluster to another HDFS cluster? Thanks, Aniket Ray On Fri, Oct 15, 2010 at 5:39 AM, FreeFoodEveryday wrote: > > We have a non-HDFS cluster (it's not a S3 cluster either). We would like to > use Hadoop distcp to copy some of

Re: Prime number of reduces vs. linear hash function

http://computinglife.wordpress.com/2008/11/20/why-do-hash-functions-use-prime-numbers/ discusses the theory in detail. On Sun, Oct 24, 2010 at 7:30 AM, Shi Yu wrote: > There is a suggestion to set the number

FUSE HDFS significantly slower

My experiment is basically opening a file in write mode and calling writes multiple times (close to 5GB data) to write to that file. Thanks for the help, aniket ray