If your application is stuck in that state, it generally means your cluster
doesn't have enough resources to start it.
In the RM logs you can see how many vcores / memory the application is
asking for, and then you can check your RM configuration to see if that's
currently available on any single
Hi Marcelo,
Thanks. I think something more subtle is happening.
I'm running a single-node cluster, so there's only 1 NM. When I executed
the exact same job the 4th time, the cluster was idle, and there was
nothing else being executed. RM currently reports that I have 6.5GB of
memory and 4 cpus av
On Tue, Jun 9, 2015 at 11:31 AM, Matt Kapilevich wrote:
> Like I mentioned earlier, I'm able to execute Hadoop jobs fine even now -
> this problem is specific to Spark.
>
That doesn't necessarily mean anything. Spark apps have different resource
requirements than Hadoop apps.
Check your RM log
Apologies, I see you already posted everything from the RM logs that
mention your stuck app.
Have you tried restarting the YARN cluster to see if that changes anything?
Does it go back to the "first few tries work" behaviour?
I run 1.4 on top of CDH 5.4 pretty often and haven't seen anything like
Hi Marcelo,
Yes, restarting YARN fixes this behavior and it again works the first few
times. The only thing that's consistent is that once Spark job submissions
stop working, it's broken for good.
On Tue, Jun 9, 2015 at 4:12 PM, Marcelo Vanzin wrote:
> Apologies, I see you already posted everyt
Does it work if you don't specify a queue?
On Tue, Jun 9, 2015 at 1:21 PM, Matt Kapilevich wrote:
> Hi Marcelo,
>
> Yes, restarting YARN fixes this behavior and it again works the first few
> times. The only thing that's consistent is that once Spark job submissions
> stop working, it's broken f
Yes! If I either specify a different queue or don't specify a queue at all,
it works.
On Tue, Jun 9, 2015 at 4:25 PM, Marcelo Vanzin wrote:
> Does it work if you don't specify a queue?
>
> On Tue, Jun 9, 2015 at 1:21 PM, Matt Kapilevich
> wrote:
>
>> Hi Marcelo,
>>
>> Yes, restarting YARN fixes
>From the RM scheduler, I see 3 applications currently stuck in the
"root.thequeue" queue.
Used Resources:
Num Active Applications: 0
Num Pending Applications: 3
Min Resources:
Max Resources:
Steady Fair Share:
Instantaneous Fair Share:
On Tue, Jun 9, 2015 at 4:30 PM, Matt Kapilevich wrote:
I've tried running a Hadoop app pointing to the same queue. Same thing now,
the job doesn't get accepted. I've cleared out the queue and killed all the
pending jobs, the queue is still unusable.
It seems like an issue with YARN, but it's specifically Spark that leaves
the queue in this state. I've
I see the other jobs SUCCEEDED without issues.
Could you snapshot the FairScheduler activity as well?
My guess it, with the single core, it is reaching a NodeManager that is
still busy with other jobs and the job ends up in a waiting state.
Does the job eventually complete?
Could you potentiall
Hi nsalian,
For some reason the rest of this thread isn't showing up here. The
NodeManager isn't busy. I'll copy/paste, the details are in there.
I've tried running a Hadoop app pointing to the same queue. Same
Hi,
Thanks for the added information. Helps add more context.
Is that specific queue different from the others?
FairScheduler.xml should have the information needed.Or if you have a
separate allocations.xml.
Something of this format:
1 mb,0vcores
9 mb,0vcores
50
0.1
No, this just a random queue name I picked when submitting the job, there's
no specific configuration for it. I am not logged in, so don't have the
default fair scheduler configuration in front of me, but I don't think
that's the problem. The cluster is completely idle, there aren't any jobs
being
Hello,
Since the other queues are fine, I reckon, there may be a limit in the max
apps or memory on this queue in particular.
I don't suspect fairscheduler limits either but on this queue we may be
seeing / hitting a maximum.
Could you try to get the configs for the queue? That should provide mor
Hi Kevin I never did. I checked for free space in the root partition, don't
think this was an issue. Now that 1.4 is officially out I'll probably give
it another shot.
On Jun 22, 2015 4:28 PM, "Kevin Markey" wrote:
> Matt: Did you ever resolve this issue? When running on a cluster or
> pseudoc
15 matches
Mail list logo
