[
https://issues.apache.org/jira/browse/HADOOP-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated HADOOP-4830:
------------------------------
Attachment: HADOOP-4830-20081229-svn.txt
Attaching a new patch incorporating the review comments. Notes on some
particular points follow.
bq. assertNTasksRunningAtSteadyState: The 5 seconds time limit brings in timing
dependencies that should be avoided if we can. Ideally if we can check that two
consecutive heartbeat cycles don't change the running counts, that should be
enough. Can we check the state of the JT or the scheduler to get this
information ?
This is replaced with a
ClusterWithCapacityScheduler.WaitTillAllTasksAreOccupied to test that all the
slots of a particular type are occupied in the cluster. And this is done by
looking at the ClusterStatus and waiting till the total number of tasks runing
becomes equal to the maximum number of slots in the cluster.
bq. TestClusterWithCapacityScheduler doesn't seem specifically needed. A lot of
the tests will exercise this and it will be very obvious if it doesn't work.
Unlike the TestControlledMapReduceJob which is a simple test that can be easily
verified for correctness with the default scheduler.
The original intention was to test invalid configuration. Moved this out, it
may be done later. Or if possible, we should move the invalid configuration
related checks into CapacitySchedulerConf itself instead of having them in
CapacityTaskScheduler.start()
Others:
- Added ClusterWithCapacityScheduler.cleanUpSchedulerConfigFile.
- Test time:
TestQueueCapacities is taking an average of slightly more than 6 1/2 minutes
for each run, excluding the build time. This is after a bit of refactoring is
done to reuse clusters across multiple tests instead of starting a new cluster
for every single test. I've tried to minimize this time as far as possible, but
an independent effort should be taken up to reduce this test time.
> Have end to end tests based on MiniMRCluster to verify that queue capacities
> are honoured.
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-4830
> URL: https://issues.apache.org/jira/browse/HADOOP-4830
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Vinod K V
> Assignee: Vinod K V
> Attachments: HADOOP-4830-20081222-svn.2, HADOOP-4830-20081229-svn.txt
>
>
> At present, we only have unit tests that make use of FakeTaskManager and that
> only test the proper functionality of capacity scheduler in isolation. Many
> issues unearthed recently proved that this is not enough and that it is
> required to have end-to-end tests so that real JT is brought into the picture
> and with that the interaction of the scheduler with JT. This issue along with
> few other related jiras should automate and replace the end-to-end tests that
> are now manually done by QA, using MiniMRCluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
