Adam Roberts created SPARK-15821:
------------------------------------
Summary: Should we use mvn -T for multithreaded Spark builds?
Key: SPARK-15821
URL: https://issues.apache.org/jira/browse/SPARK-15821
Project: Spark
Issue Type: Question
Components: Build
Reporter: Adam Roberts
Priority: Minor
With Maven we can build Spark in a multithreaded way and benefit from increased
build time performance as a result.
On a machine with eight cores, I noticed the build time reduced from 20-25
minutes to five minutes; this is by building with
mvn -T 1C -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean
package
-T 1C says that we'll use one extra thread for each core available, I've never
experienced a problem with using this option (ranging from a single cored box
to one with 192 cores available)
Should we use this for building Spark quicker or is the Jenkins job
deliberately set up such that each "executor" is needed for each pull request
and we wouldn't see an improvement anyway?
This can be discovered by checking core utilization across the farm and can
potentially reduce our build times.
Here's more information on the feature:
https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3
If this isn't suitable for the current farm then I think we should document it
for those building Spark from source
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
