I'm running zinv while compiling. It seems that MAVEN_OPTS doesn't really change much? Or perhaps I'm misunderstanding something -- grepping for java i see
root 24355 102 8.8 4687376 1350724 pts/4 Sl 16:51 11:08 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -server -Xmx2g > -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m > -Dzinc.home=/root/spark/build/zinc-0.3.5.3 -classpath > /root/spark/build/zinc-0.3.5.3/lib/compiler-interface-sources.jar:/root/spark/build/zinc-0.3.5.3/lib/incremental-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/nailgun-server.jar:/root/spark/build/zinc-0.3.5.3/lib/sbt-interface.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-compiler.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-library.jar:/root/spark/build/zinc-0.3.5.3/lib/scala-reflect.jar:/root/spark/build/zinc-0.3.5.3/lib/zinc.jar > com.typesafe.zinc.Nailgun 3030 0 > root 25151 22.0 3.2 2269092 495276 pts/4 Sl+ 16:53 1:56 > /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xms256m -Xmx512m -classpath > /opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar > -Dclassworlds.conf=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3/bin/m2.conf > -Dmaven.home=/opt/anaconda/envs/spark_build/share/apache-maven-3.3.3 > -Dmaven.multiModuleProjectDirectory=/root/spark > org.codehaus.plexus.classworlds.launcher.Launcher -DzincPort=3030 clean > package -DskipTests -Pyarn -Phive -Phive-thriftserver -Phadoop-2.4 > -Dhadoop.version=2.4.0 So the heap size is still 2g even with MAVEN_OPTS set with 4g. I noticed that within build/mvn _COMPILE_JVM_OPTS is set to 2g and this is what ZINC_OPTS is set to. --Ben On Tue, Sep 8, 2015 at 11:06 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Do you run Zinc while compiling ? > > Cheers > > On Tue, Sep 8, 2015 at 7:56 AM, Benjamin Zaitlen <quasi...@gmail.com> > wrote: > >> I'm still getting errors with 3g. I've increase to 4g and I'll report >> back >> >> To be clear: >> >> export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M >> -XX:ReservedCodeCacheSize=1024m" >> >> [ERROR] GC overhead limit exceeded -> [Help 1] >>> [ERROR] >>> [ERROR] To see the full stack trace of the errors, re-run Maven with the >>> -e switch. >>> [ERROR] Re-run Maven using the -X switch to enable full debug logging. >>> [ERROR] >>> [ERROR] For more information about the errors and possible solutions, >>> please read the following articles: >>> [ERROR] [Help 1] >>> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError >>> + return 1 >>> + exit 1 >> >> >> On Tue, Sep 8, 2015 at 10:03 AM, Sean Owen <so...@cloudera.com> wrote: >> >>> It might need more memory in certain situations / running certain >>> tests. If 3gb works for your relatively full build, yes you can open a >>> PR to change any occurrences of lower recommendations to 3gb. >>> >>> On Tue, Sep 8, 2015 at 3:02 PM, Benjamin Zaitlen <quasi...@gmail.com> >>> wrote: >>> > Ah, right. Should've caught that. >>> > >>> > The docs seem to recommend 2gb. Should that be increased as well? >>> > >>> > --Ben >>> > >>> > On Tue, Sep 8, 2015 at 9:33 AM, Sean Owen <so...@cloudera.com> wrote: >>> >> >>> >> It shows you there that Maven is out of memory. Give it more heap. I >>> use >>> >> 3gb. >>> >> >>> >> On Tue, Sep 8, 2015 at 1:53 PM, Benjamin Zaitlen <quasi...@gmail.com> >>> >> wrote: >>> >> > Hi All, >>> >> > >>> >> > I'm trying to build a distribution off of the latest in master and I >>> >> > keep >>> >> > getting errors on MQTT and the build fails. I'm running the build >>> on a >>> >> > m1.large which has 7.5 GB of RAM and no other major processes are >>> >> > running. >>> >> > >>> >> >> MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M >>> -XX:ReservedCodeCacheSize=512m" >>> >> >> ./make-distribution.sh --name continuum-custom-spark-1.5 --tgz >>> -Pyarn >>> >> >> -Phive -Phive-thriftserver -Phadoop-2.4 -Dhadoop.version=2.4.0 >>> >> > >>> >> > >>> >> > >>> >> >> INFO] Spark Project GraphX ............................... SUCCESS >>> [ >>> >> >> 33.345 s] >>> >> >> [INFO] Spark Project Streaming ............................ SUCCESS >>> >> >> [01:08 >>> >> >> min] >>> >> >> [INFO] Spark Project Catalyst ............................. SUCCESS >>> >> >> [01:39 >>> >> >> min] >>> >> >> [INFO] Spark Project SQL .................................. SUCCESS >>> >> >> [02:06 >>> >> >> min] >>> >> >> [INFO] Spark Project ML Library ........................... SUCCESS >>> >> >> [02:16 >>> >> >> min] >>> >> >> [INFO] Spark Project Tools ................................ >>> SUCCESS [ >>> >> >> 4.087 s] >>> >> >> [INFO] Spark Project Hive ................................. SUCCESS >>> >> >> [01:28 >>> >> >> min] >>> >> >> [INFO] Spark Project REPL ................................. >>> SUCCESS [ >>> >> >> 16.291 s] >>> >> >> [INFO] Spark Project YARN Shuffle Service ................. >>> SUCCESS [ >>> >> >> 13.671 s] >>> >> >> [INFO] Spark Project YARN ................................. >>> SUCCESS [ >>> >> >> 20.554 s] >>> >> >> [INFO] Spark Project Hive Thrift Server ................... >>> SUCCESS [ >>> >> >> 14.332 s] >>> >> >> [INFO] Spark Project Assembly ............................. SUCCESS >>> >> >> [03:33 >>> >> >> min] >>> >> >> [INFO] Spark Project External Twitter ..................... >>> SUCCESS [ >>> >> >> 14.208 s] >>> >> >> [INFO] Spark Project External Flume Sink .................. >>> SUCCESS [ >>> >> >> 11.535 s] >>> >> >> [INFO] Spark Project External Flume ....................... >>> SUCCESS [ >>> >> >> 19.010 s] >>> >> >> [INFO] Spark Project External Flume Assembly .............. >>> SUCCESS [ >>> >> >> 5.210 s] >>> >> >> [INFO] Spark Project External MQTT ........................ FAILURE >>> >> >> [01:10 >>> >> >> min] >>> >> >> [INFO] Spark Project External MQTT Assembly ............... SKIPPED >>> >> >> [INFO] Spark Project External ZeroMQ ...................... SKIPPED >>> >> >> [INFO] Spark Project External Kafka ....................... SKIPPED >>> >> >> [INFO] Spark Project Examples ............................. SKIPPED >>> >> >> [INFO] Spark Project External Kafka Assembly .............. SKIPPED >>> >> >> [INFO] >>> >> >> >>> >> >> >>> ------------------------------------------------------------------------ >>> >> >> [INFO] BUILD FAILURE >>> >> >> [INFO] >>> >> >> >>> >> >> >>> ------------------------------------------------------------------------ >>> >> >> [INFO] Total time: 22:55 min >>> >> >> [INFO] Finished at: 2015-09-07T22:42:57+00:00 >>> >> >> [INFO] Final Memory: 240M/455M >>> >> >> [INFO] >>> >> >> >>> >> >> >>> ------------------------------------------------------------------------ >>> >> >> [ERROR] GC overhead limit exceeded -> [Help 1] >>> >> >> [ERROR] >>> >> >> [ERROR] To see the full stack trace of the errors, re-run Maven >>> with >>> >> >> the >>> >> >> -e switch. >>> >> >> [ERROR] Re-run Maven using the -X switch to enable full debug >>> logging. >>> >> >> [ERROR] >>> >> >> [ERROR] For more information about the errors and possible >>> solutions, >>> >> >> please read the following articles: >>> >> >> [ERROR] [Help 1] >>> >> >> http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError >>> >> >> + return 1 >>> >> >> + exit 1 >>> >> > >>> >> > >>> >> > Any thoughts would be extremely helpful. >>> >> > >>> >> > --Ben >>> > >>> > >>> >> >> >
