[
https://issues.apache.org/jira/browse/SPARK-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182460#comment-15182460
]
Dongjoon Hyun commented on SPARK-12243:
---------------------------------------
According to the log, the total time of all tests are **3077s**.
So, the minimum required time for 4 processes is **769s**.
> PySpark tests are slow in Jenkins
> ---------------------------------
>
> Key: SPARK-12243
> URL: https://issues.apache.org/jira/browse/SPARK-12243
> Project: Spark
> Issue Type: Sub-task
> Components: Project Infra, PySpark, Tests
> Reporter: Josh Rosen
>
> In the Jenkins pull request builder, it looks like PySpark tests take around
> 992 seconds (~16.5 minutes) of end-to-end time to run, despite the fact that
> we run four Python test suites in parallel. We should try to figure out why
> this is slow and see if there's any easy way to speed things up.
> Note that the PySpark streaming tests take about 5 minutes to run, so
> best-case we're looking at a 10 minute speedup via further parallelization.
> We should also try to see whether there are individual slow tests in those
> Python suites which can be sped up or skipped.
> We could also consider running only the Python 2.6 tests in non-Pyspark pull
> request builds and reserve testing of all Python versions for builds which
> touch PySpark-related code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
