Applications always showing in pending state even after cluster restart

[Gaurav Chhabra]

Hi All,


I have started learning Hadoop and its related components. I am following a
tutorial on Hadoop Administration on Udemy. As part of the learning
process, i ran the following command:

$ hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jarrandomtextwriter
-Ddfs.replication=1 /user/bigdata/randomtextwriter

Above command created 30 files each of size 1 GB. Then i ran the below
reduce command:

$ yarn jar/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.input.fileinputformat.split.minsize=268435456\
-Dmapreduce.job.reduces=8 \
/user/bigdata/randomtext \
/user/bigdata/wordcount

After executing the above command, I just thought of killing the
application after some time so i ran 'yarn application -list' first which
listed a lot many applications out of which one was *wordc**ount*. I killed
that particular application using 'yarn application -kill application-id'.
However, when i checked the scheduler, i could see that several
applications were still showing in Pending state so i ran the following
command:

$ for x in $(yarn application -list -appStates ACCEPTED | awk 'NR > 2 {
print $1 }'); do yarn application -kill $x; done

It was killing the applications as I could see the 'Apps Completed' count
was going up but as soon as all the apps got killed, I saw those
applications again getting created. Even if I stop the whole cluster and
start again, the scheduler shows that there are submitted applications in
Pending state.

Here's the content of fair-scheduler.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<allocations>
    <queue name="root">
        <schedulingPolicy>drf</schedulingPolicy>
        <queue name="default">
            <schedulingPolicy>drf</schedulingPolicy>
        </queue>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false"/>
        <rule name="default" create="true"/>
    </queuePlacementPolicy>
</allocations>

This is just a test cluster.  I just want to kill the applications/clear
the application queue. Any help will really be appreciated as I am
struggling with it for the last few days.


Regards

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org