[
https://issues.apache.org/jira/browse/HADOOP-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated HADOOP-4830:
------------------------------
Attachment: HADOOP-4830-20090106-2-svn.txt
Attaching a new patch. Incorporated the above review comments. Notes about
particular points follow:
bq. We are iterating over the task list to get the number of running tasks in
ControlledMapReduceJob.getRunningTasksCount(). We check if the task is running
using TaskInProgress.isRunning(). This method of computation seems like it
would not be different from JobInProgress.runningMaps() or
JobInProgress.runningReduces(). Can you please check if there is a difference ?
As pointed out, there is no real difference. The original intention was to
ensure that so and so number of tasks are really running on TaskTrackers. But
on retrospection, I came to conclude that the number of tasks
scheduled(JIP.runningTasks()) should suffice. This is because the number of
tasks scheduled is same as the number of tasks running in the test
environment(no lost trackers). Made changes to use JIP.runningTasks() regarding
the same.
bq. This may be that we need to fix something in ivy.xml of the capacity
scheduler. Copying the ivy.xml from streaming ran the test successfully.
Made changes to ivy.xml in capacity scheduler src to include the needed jars.
But, as Hemanth also concurs during a discussion, it may become cumbersome in
future to add every new jar that capacity scheduler might not need directly but
still needed to be included because of the dependency on underlying
projects/modules say mapred/hdfs. Will file a new issue to see if this can be
addressed generally.
While running tests, found that some tests were timing out. The actual reason
turned out to be HADOOP-4977. The test TestQueueCapacities might fail sometimes
because of that and till that gets fixed.
ant test-patch results:
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath
integrity.
{code}
> Have end to end tests based on MiniMRCluster to verify that queue capacities
> are honoured.
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-4830
> URL: https://issues.apache.org/jira/browse/HADOOP-4830
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Vinod K V
> Assignee: Vinod K V
> Attachments: HADOOP-4830-20081222-svn.2,
> HADOOP-4830-20081229-svn.txt, HADOOP-4830-20090106-2-svn.txt
>
>
> At present, we only have unit tests that make use of FakeTaskManager and that
> only test the proper functionality of capacity scheduler in isolation. Many
> issues unearthed recently proved that this is not enough and that it is
> required to have end-to-end tests so that real JT is brought into the picture
> and with that the interaction of the scheduler with JT. This issue along with
> few other related jiras should automate and replace the end-to-end tests that
> are now manually done by QA, using MiniMRCluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
