[ https://issues.apache.org/jira/browse/SPARK-10372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or updated SPARK-10372: ------------------------------ Component/s: (was: Spark Core) Tests Scheduler > Add end-to-end tests for the scheduling code > -------------------------------------------- > > Key: SPARK-10372 > URL: https://issues.apache.org/jira/browse/SPARK-10372 > Project: Spark > Issue Type: Sub-task > Components: Scheduler, Tests > Affects Versions: 1.5.0 > Reporter: Imran Rashid > Assignee: Imran Rashid > > The current testing framework for the scheduler only tests individual classes > in isolation: {{DAGSchedulerSuite}}, {{TaskSchedulerImplSuite}}, etc. Of > course that is useful, but we are missing tests which cover the interaction > between these components. We also have larger tests which run entire spark > jobs, but that doesn't allow fine grained control of failures for verifying > spark's fault-tolerance. > Adding a framework for testing the scheduler as a whole will: > 1. Allow testing bugs which involve the interaction between multiple parts of > the scheduler, eg. SPARK-10370 > 2. Greater confidence in refactoring the scheduler as a whole. Given the > tight coordination between the components its hard to consider any > refactoring, since it would be unlikely to be covered by any tests. > 3. Make it easier to increase test coverage. Writing tests for the > {{DAGScheduler}} now requires intimate knowledge of exactly how the > components fit together -- a lot of work goes into mimicking the appropriate > behavior of the other components. Furthermore, it makes the tests harder to > understand for the un-initiated -- which parts are simulating some condition > of an external system (eg., losing an executor), and which parts are just > interaction with other parts of the scheduler (eg., task resubmission)? > These tests will allow to just work at the level of the interaction w/ the > executors -- tasks complete, tasks fail, executors are lost, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org