Y’all may already know this, but I haven’t seen it mentioned anywhere in our docs on here and it’s a pretty easy win.
Maven supports parallel builds <https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3> with the -T command line option. For example: ./build/mvn -T 1C -Dhadoop.version=1.2.1 -DskipTests clean package This will have Maven use 1 thread per core on your machine to build Spark. On my little MacBook air, this cuts the build time from 14 minutes to 10.5 minutes. A machine with more cores should see a bigger improvement. Note though that the docs mark this as experimental, so I wouldn’t change our reference build to use this. But it should be useful, for example, in Jenkins or when working locally. Nick
