[
https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143559#comment-14143559
]
Josh Rosen commented on SPARK-3431:
-----------------------------------
It would be great to address this soon, since several open PRs plan to add
expensive new test suites (Hive integration tests, Selenium tests for the web
UI, etc.).
There are some thread-safety issues when running multiple SparkContexts in the
same JVM, so for now we're restricted to running one test suite per JVM.
However, I think we should be able to parallelize the execution of tests from
different subprojects, e.g. by running Spark SQL tests in parallel with Spark
Streaming tests (each using its own JVM).
Our Jenkins cluster is pretty underutilized, so I don't think this will cause
problems. We also recently increased the file descriptor ulimits, so this
shouldn't cause any issues with port exhaustion, etc.
> Parallelize execution of tests
> ------------------------------
>
> Key: SPARK-3431
> URL: https://issues.apache.org/jira/browse/SPARK-3431
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Nicholas Chammas
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common
> strategy to cut test time down is to parallelize the execution of the tests.
> Doing that may in turn require some prerequisite changes to be made to how
> certain tests run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
