Hi Sean, Thanks for checking this.
I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark " You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3" <https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3>. More detailed documentation is available from the project site, at "Building Spark" <https://spark.apache.org/docs/latest/building-spark.html>. " This used to work while building older version of spark(2.4.3, 2.3.2 etc). build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package Also I have seen the maven version is changed from 3.5.4 to 3.6.3 in master branch compared to spark 2.4.3. Not sure if it's due to some bug in maven version used in master or some new change added in the master branch that prevent the parallel build. Regards Saurabh Chawla On Sat, Jan 18, 2020 at 2:19 AM Sean Owen <sro...@gmail.com> wrote: > I don't believe you can use a parallel build indeed. Some things > collide with each other. Some of the suites are run in parallel inside > the build though already. > > On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <s.saurabh...@gmail.com> > wrote: > > > > Hi All, > > > > Spark master build hangs using parallel build option in maven. On > running build the sequentially on spark master using maven, build did not > hang. This issue occurs on giving hadoop-provided (-Phadoop-provided > -Dhadoop.version=2.8.5) option. Same command works fine to build > spark-2.4.3 parallelly > > > > Command to build spark master sequentially - Spark build works fine > > build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false > -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver > -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 > -DskipTests=true clean package > > > > Command to build spark master parallel - spark build hangs > > build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false > -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver > -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 > -DskipTests=true -T 4 clean package > > > > This is the trace which keeps on repeating in maven logs > > > > [DEBUG] building maven31 dependency graph for > org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with > Maven31DependencyGraphBuilder > > [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, > ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, > ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, > ConflictIdSorter.conflictIdCount=105, > ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, > ConflictResolver.conflictItemCount=193, > DefaultDependencyCollector.collectTime=1020759, > DefaultDependencyCollector.transformTime=775495} > > [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT > > [DEBUG] > org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile > > [DEBUG] > org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile > > [DEBUG] io.netty:netty-all:jar:4.1.42.Final:compile (version > managed from 4.1.42.Final) > > [DEBUG] org.apache.commons:commons-lang3:jar:3.9:compile > (version managed from 3.9) > > [DEBUG] > org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed > from 1.8) > > [DEBUG] > com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version > managed from 2.10.0) > > [DEBUG] > com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version > managed from 2.10.0) > > [DEBUG] > com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version > managed from 2.10.0) > > [DEBUG] com.google.code.findbugs:jsr305:jar:3.0.0:compile > (version managed from 3.0.0) > > [DEBUG] com.google.guava:guava:jar:14.0.1:provided (scope > managed from compile) (version managed from 14.0.1) > > [DEBUG] org.apache.commons:commons-crypto:jar:1.0.0:compile > (version managed from 1.0.0) (exclusions managed from > [net.java.dev.jna:jna:*:*]) > > [DEBUG] io.dropwizard.metrics:metrics-core:jar:4.1.1:compile > (version managed from 4.1.1) > > [DEBUG] org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test > > [DEBUG] org.scala-lang:scala-library:jar:2.12.10:compile (version > managed from 2.12.10) > > [DEBUG] org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test > > [DEBUG] org.apache.hadoop:hadoop-client:jar:2.8.5:provided > (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, > asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, > org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, > commons-beanutils:commons-beanutils-core:*:*, > commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, > org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, > junit:junit:*:*, com.sun.jersey:*:*:*, > com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, > net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, > org.eclipse.jetty:jetty-webapp:*:*]) > > [DEBUG] org.apache.hadoop:hadoop-common:jar:2.8.5:provided > > [DEBUG] com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided > > [DEBUG] commons-cli:commons-cli:jar:1.2:provided > > [DEBUG] org.apache.commons:commons-math3:jar:3.4.1:provided > (version managed from 3.1.1) > > [DEBUG] org.apache.httpcomponents:httpclient:jar:4.5.6:provided > (version managed from 4.5.2) > > [DEBUG] > org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed > from 4.4.10) > > [DEBUG] commons-codec:commons-codec:jar:1.10:provided (version > managed from 1.11) > > [DEBUG] commons-io:commons-io:jar:2.4:provided (version managed > from 2.5) > > [DEBUG] commons-net:commons-net:jar:3.1:provided (version > managed from 3.6) > > [DEBUG] > commons-collections:commons-collections:jar:3.2.2:provided (version managed > from 3.2.2) > > [DEBUG] > org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope > managed from compile) (version managed from 9.3.19.v20170502) > > [DEBUG] > org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope > managed from compile) (version managed from 9.4.18.v20190429) > > [DEBUG] javax.servlet.jsp:jsp-api:jar:2.1:provided > > [DEBUG] log4j:log4j:jar:1.2.17:provided (scope managed from > compile) (version managed from 1.2.17) > > [DEBUG] commons-lang:commons-lang:jar:2.6:provided (version > managed from 2.6) > > [DEBUG] commons-beanutils:commons-beanutils:jar:1.9.4:provided > (version managed from 1.9.3) > > [DEBUG] > org.apache.commons:commons-configuration2:jar:2.1.1:provided > > [DEBUG] org.apache.avro:avro:jar:1.8.2:provided (version > managed from 1.7.7) > > [DEBUG] > org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed > from compile) (version managed from 1.9.13) > > [DEBUG] com.thoughtworks.paranamer:paranamer:jar:2.8:runtime > (scope managed from compile) (version managed from 2.7) > > [DEBUG] org.xerial.snappy:snappy-java:jar:1.1.7.3:provided > (version managed from 1.1.1.3) > > [DEBUG] org.tukaani:xz:jar:1.5:provided > > [DEBUG] com.google.re2j:re2j:jar:1.1:provided > > [DEBUG] com.google.protobuf:protobuf-java:jar:2.5.0:provided > (scope managed from compile) (version managed from 2.5.0) > > [DEBUG] com.google.code.gson:gson:jar:2.2.4:provided > > [DEBUG] org.apache.hadoop:hadoop-auth:jar:2.8.5:provided > > [DEBUG] com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided > > [DEBUG] > com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided > > [DEBUG] net.minidev:json-smart:jar:2.3:provided > > [DEBUG] net.minidev:accessors-smart:jar:1.2:provided > > [DEBUG] > org.apache.curator:curator-framework:jar:2.7.1:provided (version managed > from 2.12.0) > > [DEBUG] org.apache.curator:curator-client:jar:2.7.1:provided > (version managed from 2.12.0) > > [DEBUG] org.apache.curator:curator-recipes:jar:2.7.1:provided > (scope managed from compile) (version managed from 2.12.0) (exclusions > managed from []) > > [DEBUG] > org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided > > [DEBUG] org.apache.commons:commons-compress:jar:1.4.1:provided > > [DEBUG] org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-client:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerby-config:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-core:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerby-pkix:jar:1.0.1:provided > > [DEBUG] > org.apache.kerby:kerby-asn1:jar:1.0.1:provided > > [DEBUG] > org.apache.kerby:kerby-util:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-common:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-crypto:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-util:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:token-provider:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-admin:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerb-server:jar:1.0.1:provided > > [DEBUG] > org.apache.kerby:kerb-identity:jar:1.0.1:provided > > [DEBUG] org.apache.kerby:kerby-xdr:jar:1.0.1:provided > > [DEBUG] org.codehaus.woodstox:stax2-api:jar:3.1.4:provided > > [DEBUG] com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided > > [DEBUG] org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided > > [DEBUG] com.squareup.okhttp:okhttp:jar:2.7.5:provided > > [DEBUG] com.squareup.okio:okio:jar:1.6.0:provided > > [DEBUG] org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided > (scope managed from compile) (version managed from 2.8.5) (exclusions > managed from [org.apache.hadoop:hadoop-annotations:*:*, > com.google.inject:guice:*:*, > com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, > com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, > com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, > com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, > io.netty:netty:*:*]) > > [DEBUG] javax.xml.bind:jaxb-api:jar:2.2.11:provided > > [DEBUG] > com.googlecode.json-simple:json-simple:jar:1.1.1:provided > > [DEBUG] org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided > (scope managed from compile) (version managed from 2.8.5) (exclusions > managed from [org.apache.hadoop:hadoop-yarn-api:*:*, > org.apache.hadoop:hadoop-yarn-common:*:*, > org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, > commons-cli:commons-cli:*:*, log4j:log4j:*:*, > com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, > com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, > io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*]) > > [DEBUG] > org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided > > [DEBUG] org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided > (scope managed from compile) (version managed from 2.8.5) (exclusions > managed from []) > > [DEBUG] javax.servlet:javax.servlet-api:jar:3.1.0:provided > > [DEBUG] > org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed > from compile) (version managed from 9.3.19.v20170502) > > [DEBUG] > > com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided > (version managed from 2.7.8) > > [DEBUG] > jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided > > [DEBUG] > jakarta.activation:jakarta.activation-api:jar:1.2.1:provided > > [DEBUG] > com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided > > [DEBUG] > com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided > > [DEBUG] > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided > > [DEBUG] > org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided > > [DEBUG] org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided > > [DEBUG] org.slf4j:slf4j-api:jar:1.7.16:provided > > [DEBUG] org.spark-project.spark:unused:jar:1.0.0:compile > > [DEBUG] org.scalatest:scalatest_2.12:jar:3.0.8:test > > [DEBUG] org.scalactic:scalactic_2.12:jar:3.0.8:test > > [DEBUG] org.scala-lang:scala-reflect:jar:2.12.10:test (version > managed from 2.12.8) > > [DEBUG] org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test > (version managed from 1.2.0) > > [DEBUG] junit:junit:jar:4.12:test > > [DEBUG] org.hamcrest:hamcrest-core:jar:1.3:test (scope managed > from compile) (version managed from 1.3) > > [DEBUG] com.novocode:junit-interface:jar:0.11:test > > [DEBUG] org.scala-sbt:test-interface:jar:1.0:test > > [DEBUG] updateExcludesInDeps() > > > > I'll be very grateful if someone could help. > > > > Regards > > Saurabh Chawla > > >
