Github user chenqin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r207070253
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierCoordinator.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software
Github user chenqin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r207071402
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -359,17 +368,42 @@ private[spark] class TaskSchedulerImpl(
Github user jiangxb1987 closed the pull request at:
https://github.com/apache/spark/pull/21494
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193953345
--- Diff: core/src/main/scala/org/apache/spark/util/RpcUtils.scala ---
@@ -44,7 +44,7 @@ private[spark] object RpcUtils {
/** Returns the default
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193953432
--- Diff: core/src/main/scala/org/apache/spark/barrier/BarrierRDD.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r194172809
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -123,6 +124,21 @@ private[spark] class TaskSetManager(
//
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r194176991
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierCoordinator.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r194168480
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -179,6 +182,21 @@ private[spark] abstract class
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193649314
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -359,17 +368,42 @@ private[spark] class TaskSchedulerImpl(
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193648185
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1310,6 +1311,44 @@ class DAGScheduler(
}
}
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193647168
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierCoordinator.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193658009
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierCoordinator.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193640783
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierTaskContext.scala ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193644506
--- Diff:
core/src/main/scala/org/apache/spark/barrier/BarrierCoordinator.scala ---
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193290266
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193269255
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite with
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193289530
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193291076
--- Diff: python/pyspark/worker.py ---
@@ -232,6 +236,13 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193555968
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -123,6 +124,21 @@ private[spark] class TaskSetManager(
// TODO: We
Github user galv commented on a diff in the pull request:
https://github.com/apache/spark/pull/21494#discussion_r193269297
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -627,6 +627,52 @@ class SparkContextSuite extends SparkFunSuite with
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/21494
[WIP][SPARK-24375][Prototype] Support barrier scheduling
## What changes were proposed in this pull request?
Add new RDDBarrier and BarrierTaskContext to support barrier scheduling in
21 matches
Mail list logo
