Reduce > copy at 0.00 MB/s
Hey, Can anyone explain me what is reduce > copy phase in the reducer section ? The (K,List(V)), is passed to the reducer. Is reduce > copy representing copying of K,List(V) on the reducer from all mappers ? I am monitoring my jobs on the cluster, using Jobtracker url. I am seeing for most of my reducing jobs, something like this : task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at 0.00 MB/s) > 25-Jan-2012 03:54:06 task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at 0.00 MB/s) > 25-Jan-2012 03:54:06 task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at 0.00 MB/s) > 25-Jan-2012 03:54:06 Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. Is it because this reduce > copy is happening on the same machine. Thanks, Praveenesh
Re: Reduce > copy at 0.00 MB/s
i face the same issue but after sumtime when i balanced the cluster the jobs started running fine, On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar wrote: > Hey, > > Can anyone explain me what is reduce > copy phase in the reducer section ? > The (K,List(V)), is passed to the reducer. Is reduce > copy representing > copying of K,List(V) on the reducer from all mappers ? > > I am monitoring my jobs on the cluster, using Jobtracker url. > I am seeing for most of my reducing jobs, something like this : > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > Is it because this reduce > copy is happening on the same machine. > > Thanks, > Praveenesh >
Re: Reduce > copy at 0.00 MB/s
@hadoophive Can you explain more by "balance the cluster" ? Thanks, Praveenesh On Wed, Jan 25, 2012 at 4:29 PM, hadoop hive wrote: > i face the same issue but after sumtime when i balanced the cluster the > jobs started running fine, > > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar >wrote: > > > Hey > > > > Can anyone explain me what is reduce > copy phase in the reducer section > ? > > The (K,List(V)), is passed to the reducer. Is reduce > copy representing > > copying of K,List(V) on the reducer from all mappers ? > > > > I am monitoring my jobs on the cluster, using Jobtracker url. > > I am seeing for most of my reducing jobs, something like this : > > > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > > Is it because this reduce > copy is happening on the same machine. > > > > Thanks, > > Praveenesh > > >
Re: Reduce > copy at 0.00 MB/s
this problem arise after adding a node , so then i start balancer to make it balance , On Wed, Jan 25, 2012 at 4:38 PM, praveenesh kumar wrote: > @hadoophive > > Can you explain more by "balance the cluster" ? > > Thanks, > Praveenesh > > On Wed, Jan 25, 2012 at 4:29 PM, hadoop hive wrote: > > > i face the same issue but after sumtime when i balanced the cluster the > > jobs started running fine, > > > > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar > >wrote: > > > > > Hey > > > > > > Can anyone explain me what is reduce > copy phase in the reducer > section > > ? > > > The (K,List(V)), is passed to the reducer. Is reduce > copy > representing > > > copying of K,List(V) on the reducer from all mappers ? > > > > > > I am monitoring my jobs on the cluster, using Jobtracker url. > > > I am seeing for most of my reducing jobs, something like this : > > > > > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > > > > > > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > > > Is it because this reduce > copy is happening on the same machine. > > > > > > Thanks, > > > Praveenesh > > > > > >
Re: Reduce > copy at 0.00 MB/s
The copy phase fetches the map outputs. It may hang for a while if there are no newly completed map outputs to fetch yet. You can raise your reducers' slowstart value to have it not spend so many cycles waiting but rather start at 80-90% of map completions, instead of default 5%. This helps your MR performance overall, if you run multiple jobs at a time, as the reduce slots aren't wasted. On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar wrote: > Hey, > > Can anyone explain me what is reduce > copy phase in the reducer section ? > The (K,List(V)), is passed to the reducer. Is reduce > copy representing > copying of K,List(V) on the reducer from all mappers ? > > I am monitoring my jobs on the cluster, using Jobtracker url. > I am seeing for most of my reducing jobs, something like this : > > task_201201250352_0001_r_00 31.05% reduce > copy (395 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > task_201201250352_0001_r_01 30.73% reduce > copy (391 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > task_201201250352_0001_r_02 30.89% reduce > copy (393 of 424 at 0.00 > MB/s) > 25-Jan-2012 03:54:06 > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > Is it because this reduce > copy is happening on the same machine. > > Thanks, > Praveenesh -- Harsh J Customer Ops. Engineer, Cloudera
Re: Reduce > copy at 0.00 MB/s
Yeah , I am doing it, currently its on 20 %, I guess I have to raise it more. Funny thing is, its still happening after map is 100% completed. when map is completed, it should not wait, right. But I see it still give same message, for some time. Thanks, Praveenesh On Wed, Jan 25, 2012 at 7:29 PM, Harsh J wrote: > The copy phase fetches the map outputs. It may hang for a while if > there are no newly completed map outputs to fetch yet. > > You can raise your reducers' slowstart value to have it not spend so > many cycles waiting but rather start at 80-90% of map completions, > instead of default 5%. This helps your MR performance overall, if you > run multiple jobs at a time, as the reduce slots aren't wasted. > > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar > wrote: > > Hey, > > > > Can anyone explain me what is reduce > copy phase in the reducer section > ? > > The (K,List(V)), is passed to the reducer. Is reduce > copy representing > > copying of K,List(V) on the reducer from all mappers ? > > > > I am monitoring my jobs on the cluster, using Jobtracker url. > > I am seeing for most of my reducing jobs, something like this : > > > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at > 0.00 > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > > Is it because this reduce > copy is happening on the same machine. > > > > Thanks, > > Praveenesh > > > > -- > Harsh J > Customer Ops. Engineer, Cloudera >
Re: Reduce > copy at 0.00 MB/s
Hey , Can any1 help me with this, i have increases the reduce slowstart to .25 but its still hangs after copy . tell me what else i can change it to make it working fine. regards Vikas Srivastava On Wed, Jan 25, 2012 at 7:45 PM, praveenesh kumar wrote: > Yeah , I am doing it, currently its on 20 %, I guess I have to raise it > more. > Funny thing is, its still happening after map is 100% completed. > when map is completed, it should not wait, right. But I see it still give > same message, for some time. > > Thanks, > Praveenesh > > On Wed, Jan 25, 2012 at 7:29 PM, Harsh J wrote: > > > The copy phase fetches the map outputs. It may hang for a while if > > there are no newly completed map outputs to fetch yet. > > > > You can raise your reducers' slowstart value to have it not spend so > > many cycles waiting but rather start at 80-90% of map completions, > > instead of default 5%. This helps your MR performance overall, if you > > run multiple jobs at a time, as the reduce slots aren't wasted. > > > > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar > > wrote: > > > Hey, > > > > > > Can anyone explain me what is reduce > copy phase in the reducer > section > > ? > > > The (K,List(V)), is passed to the reducer. Is reduce > copy > representing > > > copying of K,List(V) on the reducer from all mappers ? > > > > > > I am monitoring my jobs on the cluster, using Jobtracker url. > > > I am seeing for most of my reducing jobs, something like this : > > > > > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > > > > > > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > > > Is it because this reduce > copy is happening on the same machine. > > > > > > Thanks, > > > Praveenesh > > > > > > > > -- > > Harsh J > > Customer Ops. Engineer, Cloudera > > >
RE: Reduce > copy at 0.00 MB/s
Can somebody help me to unsubscribe. Even after unsubscribing I continue to get mails. -Original Message- From: hadoop hive [mailto:hadooph...@gmail.com] Sent: 02 February 2012 12:29 To: common-user@hadoop.apache.org Subject: Re: Reduce > copy at 0.00 MB/s Hey , Can any1 help me with this, i have increases the reduce slowstart to .25 but its still hangs after copy . tell me what else i can change it to make it working fine. regards Vikas Srivastava On Wed, Jan 25, 2012 at 7:45 PM, praveenesh kumar wrote: > Yeah , I am doing it, currently its on 20 %, I guess I have to raise it > more. > Funny thing is, its still happening after map is 100% completed. > when map is completed, it should not wait, right. But I see it still give > same message, for some time. > > Thanks, > Praveenesh > > On Wed, Jan 25, 2012 at 7:29 PM, Harsh J wrote: > > > The copy phase fetches the map outputs. It may hang for a while if > > there are no newly completed map outputs to fetch yet. > > > > You can raise your reducers' slowstart value to have it not spend so > > many cycles waiting but rather start at 80-90% of map completions, > > instead of default 5%. This helps your MR performance overall, if you > > run multiple jobs at a time, as the reduce slots aren't wasted. > > > > On Wed, Jan 25, 2012 at 3:34 PM, praveenesh kumar > > wrote: > > > Hey, > > > > > > Can anyone explain me what is reduce > copy phase in the reducer > section > > ? > > > The (K,List(V)), is passed to the reducer. Is reduce > copy > representing > > > copying of K,List(V) on the reducer from all mappers ? > > > > > > I am monitoring my jobs on the cluster, using Jobtracker url. > > > I am seeing for most of my reducing jobs, something like this : > > > > > > task_201201250352_0001_r_0031.05% reduce > copy (395 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0130.73% reduce > copy (391 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > task_201201250352_0001_r_0230.89% reduce > copy (393 of 424 at > > 0.00 > > > MB/s) > 25-Jan-2012 03:54:06 > > > > > > > > > > > > > > > Can anyone explain me why the speed is 0.00 MB/s. Job is running fine. > > > Is it because this reduce > copy is happening on the same machine. > > > > > > Thanks, > > > Praveenesh > > > > > > > > -- > > Harsh J > > Customer Ops. Engineer, Cloudera > > > - No virus found in this message. Checked by AVG - www.avg.com Version: 2012.0.1913 / Virus Database: 2112/4780 - Release Date: 02/01/12
