Yes that's exactly what happens, but I would think that if data node is unavailable/unavailability of data for one of the nodes should not cause indefinite wait.. Are there any properties we can set to avoid getting into indefinite/non-deterministic outcome of a spark application?
On Thu, May 16, 2019 at 9:49 AM Akshay Bhardwaj < akshay.bhardwaj1...@gmail.com> wrote: > One of the reason that any jobs running on YARN (Spark, MR, Hive, etc) can > get stuck is if there is data unavailability issue with HDFS. > This can arise if either the Namenode is not reachable or if the > particular data block is unavailable due to node failures. > > Can you check if your YARN service can communicate with Name node service? > > Akshay Bhardwaj > +91-97111-33849 > > > On Thu, May 16, 2019 at 4:27 PM Rishi Shah <rishishah.s...@gmail.com> > wrote: > >> on yarn >> >> On Thu, May 16, 2019 at 1:36 AM Akshay Bhardwaj < >> akshay.bhardwaj1...@gmail.com> wrote: >> >>> Hi Rishi, >>> >>> Are you running spark on YARN or spark's master-slave cluster? >>> >>> Akshay Bhardwaj >>> +91-97111-33849 >>> >>> >>> On Thu, May 16, 2019 at 7:15 AM Rishi Shah <rishishah.s...@gmail.com> >>> wrote: >>> >>>> Any one please? >>>> >>>> On Tue, May 14, 2019 at 11:51 PM Rishi Shah <rishishah.s...@gmail.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> At times when there's a data node failure, running spark job doesn't >>>>> fail - it gets stuck and doesn't return. Any setting can help here? I >>>>> would >>>>> ideally like to get the job terminated or executors running on those data >>>>> nodes fail... >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Rishi Shah >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Rishi Shah >>>> >>> >> >> -- >> Regards, >> >> Rishi Shah >> > -- Regards, Rishi Shah
