PSA: Maven supports parallel builds
Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win. Maven supports parallel builds https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 with the -T command line option. For example: ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package This will have Maven use 1 thread per core on your machine to build Spark. On my little MacBook air, this cuts the build time from 14 minutes to 10.5 minutes. A machine with more cores should see a bigger improvement. Note though that the docs mark this as experimental, so I wouldn’t change our reference build to use this. But it should be useful, for example, in Jenkins or when working locally. Nick
Re: PSA: Maven supports parallel builds
I've done this in the past, but back when I wasn't using Zinc it didn't make a big difference. It's worth doing this in our jenkins environment though. - Patrick On Thu, Feb 5, 2015 at 4:52 PM, Dirceu Semighini Filho dirceu.semigh...@gmail.com wrote: Thanks Nicholas, I didn't knew this. 2015-02-05 22:16 GMT-02:00 Nicholas Chammas nicholas.cham...@gmail.com: Y'all may already know this, but I haven't seen it mentioned anywhere in our docs on here and it's a pretty easy win. Maven supports parallel builds https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 with the -T command line option. For example: ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package This will have Maven use 1 thread per core on your machine to build Spark. On my little MacBook air, this cuts the build time from 14 minutes to 10.5 minutes. A machine with more cores should see a bigger improvement. Note though that the docs mark this as experimental, so I wouldn't change our reference build to use this. But it should be useful, for example, in Jenkins or when working locally. Nick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: PSA: Maven supports parallel builds
Thanks Nicholas, I didn't knew this. 2015-02-05 22:16 GMT-02:00 Nicholas Chammas nicholas.cham...@gmail.com: Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win. Maven supports parallel builds https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3 with the -T command line option. For example: ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package This will have Maven use 1 thread per core on your machine to build Spark. On my little MacBook air, this cuts the build time from 14 minutes to 10.5 minutes. A machine with more cores should see a bigger improvement. Note though that the docs mark this as experimental, so I wouldn’t change our reference build to use this. But it should be useful, for example, in Jenkins or when working locally. Nick