spark git commit: [SPARK-11066] Update DAGScheduler's "misbehaved ResultHandler"

2015-10-15 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/branch-1.5 c27e19042 -> 63ca9f921


[SPARK-11066] Update DAGScheduler's "misbehaved ResultHandler"

Restrict tasks (of job) to only 1 to ensure that the causing Exception asserted 
for job failure is the deliberately thrown DAGSchedulerSuiteDummyException 
intended, not an UnsupportedOperationException from any second/subsequent tasks 
that can propagate from a race condition during code execution.

Author: shellberg 

Closes #9076 from 
shellberg/shellberg-DAGSchedulerSuite-misbehavedResultHandlerTest-patch-1.

(cherry picked from commit 523adc24a683930304f408d477607edfe9de7b76)
Signed-off-by: Sean Owen 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/63ca9f92
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/63ca9f92
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/63ca9f92

Branch: refs/heads/branch-1.5
Commit: 63ca9f9219eddf9902e845598800155b74fb8c84
Parents: c27e190
Author: shellberg 
Authored: Thu Oct 15 18:07:10 2015 +0100
Committer: Sean Owen 
Committed: Thu Oct 15 18:07:21 2015 +0100

--
 .../org/apache/spark/scheduler/DAGSchedulerSuite.scala | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/63ca9f92/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
--
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index ed481b1..7232970 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -850,18 +850,27 @@ class DAGSchedulerSuite
 assert(sc.parallelize(1 to 10, 2).count() === 10)
   }
 
+  /**
+   * The job will be failed on first task throwing a 
DAGSchedulerSuiteDummyException.
+   *  Any subsequent task WILL throw a legitimate 
java.lang.UnsupportedOperationException.
+   *  If multiple tasks, there exists a race condition between the 
SparkDriverExecutionExceptions
+   *  and their differing causes as to which will represent result for job...
+   */
   test("misbehaved resultHandler should not crash DAGScheduler and 
SparkContext") {
 val e = intercept[SparkDriverExecutionException] {
+  // Number of parallelized partitions implies number of tasks of job
   val rdd = sc.parallelize(1 to 10, 2)
   sc.runJob[Int, Int](
 rdd,
 (context: TaskContext, iter: Iterator[Int]) => iter.size,
-Seq(0, 1),
+// For a robust test assertion, limit number of job tasks to 1; that 
is,
+// if multiple RDD partitions, use id of any one partition, say, first 
partition id=0
+Seq(0),
 (part: Int, result: Int) => throw new DAGSchedulerSuiteDummyException)
 }
 assert(e.getCause.isInstanceOf[DAGSchedulerSuiteDummyException])
 
-// Make sure we can still run commands
+// Make sure we can still run commands on our SparkContext
 assert(sc.parallelize(1 to 10, 2).count() === 10)
   }
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-11066] Update DAGScheduler's "misbehaved ResultHandler"

2015-10-15 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/master aec4400be -> 523adc24a


[SPARK-11066] Update DAGScheduler's "misbehaved ResultHandler"

Restrict tasks (of job) to only 1 to ensure that the causing Exception asserted 
for job failure is the deliberately thrown DAGSchedulerSuiteDummyException 
intended, not an UnsupportedOperationException from any second/subsequent tasks 
that can propagate from a race condition during code execution.

Author: shellberg 

Closes #9076 from 
shellberg/shellberg-DAGSchedulerSuite-misbehavedResultHandlerTest-patch-1.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/523adc24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/523adc24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/523adc24

Branch: refs/heads/master
Commit: 523adc24a683930304f408d477607edfe9de7b76
Parents: aec4400
Author: shellberg 
Authored: Thu Oct 15 18:07:10 2015 +0100
Committer: Sean Owen 
Committed: Thu Oct 15 18:07:10 2015 +0100

--
 .../org/apache/spark/scheduler/DAGSchedulerSuite.scala | 13 +++--
 1 file changed, 11 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/523adc24/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
--
diff --git 
a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 
b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
index 697c195..5b01ddb 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
@@ -1375,18 +1375,27 @@ class DAGSchedulerSuite
 assert(sc.parallelize(1 to 10, 2).count() === 10)
   }
 
+  /**
+   * The job will be failed on first task throwing a 
DAGSchedulerSuiteDummyException.
+   *  Any subsequent task WILL throw a legitimate 
java.lang.UnsupportedOperationException.
+   *  If multiple tasks, there exists a race condition between the 
SparkDriverExecutionExceptions
+   *  and their differing causes as to which will represent result for job...
+   */
   test("misbehaved resultHandler should not crash DAGScheduler and 
SparkContext") {
 val e = intercept[SparkDriverExecutionException] {
+  // Number of parallelized partitions implies number of tasks of job
   val rdd = sc.parallelize(1 to 10, 2)
   sc.runJob[Int, Int](
 rdd,
 (context: TaskContext, iter: Iterator[Int]) => iter.size,
-Seq(0, 1),
+// For a robust test assertion, limit number of job tasks to 1; that 
is,
+// if multiple RDD partitions, use id of any one partition, say, first 
partition id=0
+Seq(0),
 (part: Int, result: Int) => throw new DAGSchedulerSuiteDummyException)
 }
 assert(e.getCause.isInstanceOf[DAGSchedulerSuiteDummyException])
 
-// Make sure we can still run commands
+// Make sure we can still run commands on our SparkContext
 assert(sc.parallelize(1 to 10, 2).count() === 10)
   }
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org