[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143724#comment-14143724
]
Sean Owen commented on SPARK-3431:
----------------------------------
It's trivial to configure Maven surefire/failsafe to execute tests in parallel.
It can parallelize by class or method, fork or not, control number of
concurrent forks as a multiple of cores, etc. For example, it's no problem to
make test classes use their own JVM, and not even reuse JVMs if you don't want.
The harder part is making the tests play nice with each other on one machine
when it comes to shared resources: files and ports, really. I think the tests
have had several passes of improvements to reliably use their own temp space,
and try to use an unused port, but this is one typical cause of test breakage.
It's not yet clear that tests don't clobber each other by trying to use the
same default Spark working dir or something.
Finally, some tests that depend on a certain sequence of random numbers may
need to be made more robust.
but the parallelization is trivial in Maven, at least.
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
