[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169087#comment-14169087
]
Sean Owen commented on SPARK-3431:
----------------------------------
I just tried parallelizing scalatest and it failed fairly spectacularly. With
forkMode=once, lots of errors pop out like:
{code}
akka.actor.InvalidActorNameException: actor name [LocalBackendActor] is not
unique!
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in
stage 11.0 failed 1 times, most recent failure: Lost task 1.0 in stage 11.0
(TID 10, localhost): java.io.IOException: PARSING_ERROR(2)
org.apache.spark.SparkException: Task not serializable
at
org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
{code}
I think it may work with even more fine-grained separation of tests into JVMs,
as evidenced by Josh's success with SBT, but scalatest doesn't support that.
(surefire does.)
With forkMode=never I see different errors:
{code}
java.net.BindException: Address already in use: Service 'SparkUI' failed after
16 retries!
{code}
Maybe the several UIs start up at much more the same time when not forking
JVMs, and some end up failing through 16 retries as so many compete to allocate
ports from 4040 onwards.
This too might be better if you could control the level of parallelism, and
again surefire does that.
So I will try to see if surefire can be used, but this probably also indicates
some more work in the tests could make them more parallel-friendly too.
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
