Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
Does anyone know what the issue could be , is there any property setting that is causing this behavior ? On Wed, Sep 17, 2014 at 3:11 PM, S.L wrote: > > I don't see any hs_err file in the current directory , so I don't think > that is the case. > > On Wed, Sep 17, 2014 at 2:21 PM, S.L wrote: > >> >> I am not sure, I running a sequence of MRV1 jobs using a bash script , >> this seems to happen in the 4th iteration consistently, how do I confirm >> this possibility? >> >> On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli < >> vino...@apache.org> wrote: >> >>> Is it possible that the client JVM is somehow getting killed while the >>> YARN application finishes as usual on the cluster in the background? >>> >>> +Vinod >>> >>> On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: >>> Hi All, I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when I submit this job YARN the application that is running in YARN is marked as complete even as on console its reported as only 58% complete . I have confirmed that its also not printing the log statements that its supposed to print when the job is actually complete . Please see the output from the job submission console below. It just stops at 58% and job history server and YARN cluster UI reports that this job has already succeeded. Can someone let me know why this is happening ? 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks. >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >> >
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
I don't see any hs_err file in the current directory , so I don't think that is the case. On Wed, Sep 17, 2014 at 2:21 PM, S.L wrote: > > I am not sure, I running a sequence of MRV1 jobs using a bash script , > this seems to happen in the 4th iteration consistently, how do I confirm > this possibility? > > On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> Is it possible that the client JVM is somehow getting killed while the >> YARN application finishes as usual on the cluster in the background? >> >> +Vinod >> >> On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: >> >>> >>>Hi All, >>> >>> I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is >>> when I submit this job YARN the application that is running in YARN is >>> marked as complete even as on console its reported as only 58% complete . I >>> have confirmed that its also not printing the log statements that its >>> supposed to print when the job is actually complete . >>> >>> Please see the output from the job submission console below. It just >>> stops at 58% and job history server and YARN cluster UI reports that this >>> job has already succeeded. Can someone let me know why this is happening ? >>> >>> 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% >>> 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% >>> 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% >>> 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% >>> 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% >>> >>> Thanks. >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > >
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
I am not sure, I running a sequence of MRV1 jobs using a bash script , this seems to happen in the 4th iteration consistently, how do I confirm this possibility? On Wed, Sep 17, 2014 at 1:34 PM, Vinod Kumar Vavilapalli wrote: > Is it possible that the client JVM is somehow getting killed while the > YARN application finishes as usual on the cluster in the background? > > +Vinod > > On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: > >> >>Hi All, >> >> I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is >> when I submit this job YARN the application that is running in YARN is >> marked as complete even as on console its reported as only 58% complete . I >> have confirmed that its also not printing the log statements that its >> supposed to print when the job is actually complete . >> >> Please see the output from the job submission console below. It just >> stops at 58% and job history server and YARN cluster UI reports that this >> job has already succeeded. Can someone let me know why this is happening ? >> >> 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% >> 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% >> 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% >> 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% >> 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% >> >> Thanks. >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
Re: Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
Is it possible that the client JVM is somehow getting killed while the YARN application finishes as usual on the cluster in the background? +Vinod On Wed, Sep 17, 2014 at 9:29 AM, S.L wrote: > >Hi All, > > I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when > I submit this job YARN the application that is running in YARN is marked as > complete even as on console its reported as only 58% complete . I have > confirmed that its also not printing the log statements that its supposed > to print when the job is actually complete . > > Please see the output from the job submission console below. It just stops > at 58% and job history server and YARN cluster UI reports that this job has > already succeeded. Can someone let me know why this is happening ? > > 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% > 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% > 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% > 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% > 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% > > Thanks. > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Map Reduce Job is reported as complete on history server while on console it shows as only half way thru
Hi All, I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when I submit this job YARN the application that is running in YARN is marked as complete even as on console its reported as only 58% complete . I have confirmed that its also not printing the log statements that its supposed to print when the job is actually complete . Please see the output from the job submission console below. It just stops at 58% and job history server and YARN cluster UI reports that this job has already succeeded. Can someone let me know why this is happening ? 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks.
Re: Job is reported as complete on history server while on console it shows as only half way thru
Doe you mean multiple application attempts on YARN? One MR job shouldn't result in multiple YARN applications. Would you please share what is shown on the JHS and RM web UIs? On Thu, Aug 28, 2014 at 2:01 PM, S.L wrote: > Hi All, > > I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when > I submit this job YARN created multiple applications for that submitted job > , and the last application that is running in YARN is marked as complete > even as on console its reported as only 58% complete . I have confirmed > that its also not printing the log statements that its supposed to print > when the job is actually complete . > > Please see the output from the job submission console below. It just stops > at 58% and job history server and YARN cluster UI reports that this job has > already succeeded. > > 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% > 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% > 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% > 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% > 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% > > Thanks. > -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Job is reported as complete on history server while on console it shows as only half way thru
Hi All, I am running a MRV1 job on Hadoop YARN 2.3.0 cluster , the problem is when I submit this job YARN created multiple applications for that submitted job , and the last application that is running in YARN is marked as complete even as on console its reported as only 58% complete . I have confirmed that its also not printing the log statements that its supposed to print when the job is actually complete . Please see the output from the job submission console below. It just stops at 58% and job history server and YARN cluster UI reports that this job has already succeeded. 4/08/28 08:36:19 INFO mapreduce.Job: map 54% reduce 0% 14/08/28 08:44:13 INFO mapreduce.Job: map 55% reduce 0% 14/08/28 08:52:16 INFO mapreduce.Job: map 56% reduce 0% 14/08/28 08:59:22 INFO mapreduce.Job: map 57% reduce 0% 14/08/28 09:07:33 INFO mapreduce.Job: map 58% reduce 0% Thanks.
