[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168038#comment-14168038
]
Josh Rosen commented on SPARK-3431:
-----------------------------------
I've been playing around with this via SBT configurations and I've come up with
something that allows multiple test suites to execute in parallel with each
suite in its own JVM (at far as I can tell):
{code}
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 01a5b20..4e84c94 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -21,6 +21,7 @@ import scala.collection.JavaConversions._
import sbt._
import sbt.Classpaths.publishTask
import sbt.Keys._
+import sbt.Tests._
import sbtunidoc.Plugin.genjavadocSettings
import org.scalastyle.sbt.ScalastylePlugin.{Settings => ScalaStyleSettings}
import com.typesafe.sbt.pom.{PomBuild, SbtPomKeys}
@@ -333,6 +334,17 @@ object Unidoc {
object TestSettings {
import BuildCommons._
+ // See http://stackoverflow.com/questions/15798341 for notes on how to fork
+ // a new JVM for each test in SBT:
+ def singleTests(tests: Seq[TestDefinition]) =
+ tests map { test =>
+ new Group(
+ name = test.name,
+ tests = Seq(test),
+ runPolicy = SubProcess(javaOptions = Seq.empty[String]))
+ }
+
+
lazy val settings = Seq (
// Fork new JVMs for tests and set Java options for those
fork := true,
@@ -352,9 +364,9 @@ object TestSettings {
testOptions += Tests.Argument(TestFrameworks.JUnit, "-v", "-a"),
// Enable Junit testing.
libraryDependencies += "com.novocode" % "junit-interface" % "0.9" % "test",
- // Only allow one test at a time, even across projects, since they run in
the same JVM
- parallelExecution in Test := false,
- concurrentRestrictions in Global += Tags.limit(Tags.Test, 1),
+ parallelExecution in Test := true,
+ testGrouping in Test <<= definedTests in Test map singleTests,
+ logBuffered in Test := true,
// Remove certain packages from Scaladoc
scalacOptions in (Compile, doc) := Seq(
"-groups",
{code}
One snag that I ran into: it seems that running multiple tests suites in
parallel in separate JVMs leads to interleaved test output, making it hard to
debug failures or hangs:
https://groups.google.com/forum/#!topic/simple-build-tool/SOq8gl4zd6E. I think
that we need to fix this issue before enabling parallel tests.
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
