I've had disk thrashing up my build times from ~30 min to ~2 hours, but never this much. I have no evidence but would suspect a DNS problem resulting in multiple DNS timeouts for each module.
Do you have a more-memory-endowed machine on this network you could try a build on to see if it's faster? thanks david jencks On Jun 22, 2011, at 1:32 PM, Russell E Glaue wrote: > On 06/22/2011 01:54 PM, Kevan Miller wrote: >> >> On Jun 22, 2011, at 1:27 PM, Russell E Glaue wrote: >> >>> On 06/22/2011 12:02 PM, Kevan Miller wrote: >>>> >>>> On Jun 22, 2011, at 12:09 PM, Russell E Glaue wrote: >> >> <snip> >> >>>> >>>>> >>>>> I am using a very basic build procedure. >>>>> 1. checkout G3 trunk >>>>> 2. JAVA_HOME="/path/to/jdk/1.6.0_22" >>>>> 3. download and install a fresh maven 2.2.1 >>>>> 4. MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m >>>>> -XX:ReservedCodeCacheSize=64m" >>>> >>>> My settings are similar. I don't specify '-XX:ReservedCodeCacheSize=64m'. >>>> >>>> I run with the following. I know there are others running with larger max >>>> heaps. I haven't : >>>> >>>> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError" > > Without the "-XX:ReservedCodeCacheSize=64m" option in MAVEN_OPTS I get this > OutOfMemory error: > - > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Geronimo Build Support :: Plugin 3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > ...<snip>... > [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ > buildsupport-maven-plugin --- > [INFO] Compiling 3 source files to > /data/geronimo/asf-geronimo-server-trunk/framework/buildsupport/buildsupport-maven-plugin/target/classes > [INFO] > [INFO] --- gmaven-plugin:1.2:compile (default) @ buildsupport-maven-plugin --- > --------------------------------------------------- > constituent[0]: > file:/usr/local/apache-maven-3.0.3/lib/maven-embedder-3.0.3.jar > ...<snip>... > constituent[30]: file:/usr/local/apache-maven-3.0.3/lib/maven-compat-3.0.3.jar > --------------------------------------------------- > Exception in thread "main" java.lang.VirtualMachineError: out of space in > CodeCache for adapters > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:212) > at > org.codehaus.groovy.ast.builder.AstBuilderTransformation.visit(AstBuilderTransformation.groovy:62) > at > org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:268) > at > org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:799) > at > org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464) > at > org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:443) > at > org.codehaus.gmaven.runtime.v1_6.ClassCompilerFeature$ClassCompilerImpl.compile(ClassCompilerFeature.java:159) > at > org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.compile(AbstractCompileMojo.java:200) > at > org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.process(AbstractCompileMojo.java:164) > at > org.codehaus.gmaven.plugin.ComponentMojoSupport.doExecute(ComponentMojoSupport.java:60) > at org.codehaus.gmaven.plugin.MojoSupport.execute(MojoSupport.java:69) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > - > > With the "-XX:ReservedCodeCacheSize=64m" flag, I do not get this error. > > >>> >>> The server I am build on has 1024MB of memory total, and has a dual >>> dual-core >>> Intel(R) XEON(TM) CPU 2.00GHz with 512KB cache. >>> >>> I have tried many JVM configurations, and the one like mine and yours is the >>> best one I have found. For me, a JVM configuration with less than 400m for >>> MaxPermSize always ended with an OutOfMemory error on PermGen space. >>> >>> The Geronimo dev wiki suggests 200m for MaxPermSize, which never worked for >>> me >>> for trunk. I always assumed it was because of all the snapshot downloading >>> the >>> build process has to do. Even after the 2nd or 3rd build, 200m was never >>> enough. >> >> Right. So, MaxPermSize=200m is definitely out-of-date. There is/may be some >> bugs that are increasing this value. However, not really worth our while to >> pursue at the moment. I don't think MaxPermSize=500m is too prohibitive. >> More important things to do, ATM. >> >> That said -- I'm building on a machine with 8 gigs of real memory. So, >> 1024MB of real memory seems very small to me. And I suspect that this is the >> source of your problem... A max heap of 1024MB and MaxPermSize of 512m means >> your system may be spending a lot of time swapping virtual memory. > > So wrote what are your JVM Options set at: > MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError" > Mine are: > MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m" > > The machine I am using has 1024MB of real memory and is dedicated to building > geronimo (and it is not a virtual server). I set the heap max to 968m to keep > the memory usage from peaking at 100% of system capacity. > > But there is a chance you may be right about the swaping. the machine runs at > 100% real memory usage (expected) and about ~50% swap. > > > I thinking recommending a minimum amount of real memory a machine should have > to > adequately compile G3 in a reasonable amount of time should be suggested. > What would you say the minimum recommendation should be? For Windows and > Linux? > 2GB for Windows and 1.5GB for Linux, assuming the machine is dedicated to > compiling? > >> >>> >>>> >>>>> 5. mvn clean install >>>>> 6. 13 to 39 hours later == BUILD SUCCESS >>>>> >>>>> >>>>> From observing the build process, it would seem the most used (seemingly >>>>> paused) >>>>> time is right before the build process moves on to the next plugin or >>>>> assembly. >>>>> >>>>> This "seemingly paused time" after the assembly I assume is the packaging >>>>> process, because the resulting output is a huge collection. >>>>> >>>>> [INFO] Geronimo Assemblies ............................... SUCCESS >>>>> [3.995s] >>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS >>>>> [8:15.210s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS >>>>> [1:57:17.311s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS >>>>> [1:02:47.374s] >>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS >>>>> [8:08.838s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS >>>>> [1:59:16.619s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>> [1:08:13.854s] >>>>> >>>>> That is ~1.5 hours for the "Java EE 6 Web Profile + Jetty8" and "Java EE >>>>> 6 Web >>>>> Profile + Tomcat7" assemblies. >>>> >>>> Here are my build times. This is running without tests. Which would affect >>>> the overall build time. Assembly times would be unaffected -- we don't run >>>> any tests when building assemblies. >>>> >>>> [INFO] Geronimo Assemblies ............................... SUCCESS [0.163s] >>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS >>>> [42.253s] >>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS >>>> [1:04.295s] >>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS >>>> [1:33.097s] >>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS >>>> [41.587s] >>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS >>>> [1:14.409s] >>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>> [1:46.333s] >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] BUILD SUCCESS >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Total time: 28:15.199s >>>> [INFO] Finished at: Wed Jun 22 12:38:10 EDT 2011 >>>> [INFO] Final Memory: 633M/1011M >>>> [INFO] >>>> ------------------------------------------------------------------------ >>> >>> Yours is about 1/3 faster than mine. >> >> I think you've missed the metric. My times are an order of magnitude (hours >> vs minutes) better than yours. >> >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>> [1:08:13.854s] >> >> and >> >>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>> [1:46.333s] >> >> That's 68 minutes vs 1 minute 46 seconds. My *total* build time was 28 >> minutes. > > Oh. ya. Let me rub my eyes and take another look there.... your right! > >> >>> But you might have a faster processor too. >>> Obviously, this part of the build is not the time consumer for me. It is >>> nice to >>> know this part of the build is supposed to take several hours - that helps >>> me >>> narrow down the issues. > > Okay - this was an incorrect statement by me. > >>> >>>> >>>> >>>>> >>>>> >>>>> For the plugins, I would assume the same, except that I notice that, of >>>>> course, >>>>> the latest snapshot dependencies are being downloaded, and the poms in >>>>> the maven >>>>> SNAPSHOT repositories are continually being accessed. >>>>> >>>>> I realize that because trunk (G3) largely depends on 3rd-party artifact >>>>> SNAPSHOTS that these repositories are checked during every build, and new >>>>> SNAPSHOT libraries are always being downloaded (as is the nature of >>>>> snapshot >>>>> releases). >>>> >>>> Correct. A local maven proxy (e.g. Nexus) can reduce the overhead. >>> >>> So it is worth noting as it already is now. >>> >>>> >>>>> >>>>> As it is not a simple process for a user to setup a maven repo proxy, and >>>>> could >>>>> be a deterrent to someone getting started if it were necessary to install >>>>> a >>>>> proxy, I am looking at other methods and tips that might increase the >>>>> speed of >>>>> building G3. >>>> >>>> I've tried running with a local maven proxy. However, I've found it to be >>>> too much trouble to maintain/configure. >>> >>> Agreed. At least it is too much trouble if you are not continuously >>> building the >>> entire trunk every day on the larger scale. So too much to ask a G3 >>> beginner to do. >> >> Too much to ask of a new maven user -- I agree. >> >>> >>>> >>>>> >>>>> Is this feasible? Or would we just say, that because of the nature of >>>>> SNAPSHOT >>>>> builds and their dependencies on 3rd-party SNAPSHOT artifacts, there is >>>>> no real >>>>> means to improving build speed unless a proxy is utilized? >>>>> If the later is true, I would like to add this note in the wiki's build >>>>> documentation. And also warn those who choose to build without a proxy may >>>>> experience a long build time. >>>> >>>> While I'm thinking about it -- what is your network environment? The focus >>>> on proxies imply that your slow build times are predominantly network >>>> issue. Are you certain that is the case? >>> >>> We have about ~14Mb/s speed to the Internet. Our backplane is Gigabit >>> Ethernet. >>> We are connected to a major Internet backbone provider. >>> I have been wondering if the problem is responsiveness of the maven repo >>> sites. >>> I have been seeing on occasion a refusal to download an artifact from the >>> site, >>> where as I was able to download it within a browser from the same URL. >>> So I wonder if the repo sites are either metering the usage on a per user or >>> session basis. >> >> I'm building on my home broadband connection. Probably similar speeds. I >> think we should identify the source of your extremely slow build time. >> >> If remote repository lookups are the cause of your problems, 'mvn -o clean >> install' should result in a much faster build time. However, I suspect that >> it won't make much difference at all... > > It seems a lot of "pauses" in the build process (while using the > '-DskipTests=true' flag) is at the end of building each plugin, and that looks > like this (example plugin build output)... > (Note: almost every plugin gives the log4j errors) > - > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Geronimo Plugins, Corba :: Deployer 3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > [INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ > openejb-corba-deployer --- > [INFO] > [INFO] --- genesis-maven-plugin:2.0:validate-configuration (default) @ > openejb-corba-deployer --- > [INFO] > [INFO] --- geronimo-property-plugin:3.0-SNAPSHOT:set-property (set-property) @ > openejb-corba-deployer --- > [INFO] > [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (default) @ > openejb-corba-deployer --- > [INFO] > [INFO] --- maven-remote-resources-plugin:1.0:process (default) @ > openejb-corba-deployer --- > > !!!!! somewhat of a pause here, can be up to 60 seconds sometimes !!!!! > > [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference > : > template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is > not a > valid reference. > [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference > : > template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is > not a > valid reference. > [INFO] > [INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ > openejb-corba-deployer --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] skip non existing resourceDirectory > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/resources > [INFO] skip non existing resourceDirectory > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/filtered-resources > [INFO] Copying 3 resources > > !!!!! for the plugins that use the maven-compiler-plugin:2.0.2:compile, of > course this will take time as it is compiling !!!!! > !!!!! However, the CPU usage only rises for like the first quarter of this > wait > time. !!!!! > > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:validate-configuration > (default-validate-configuration) @ openejb-corba-deployer --- > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-plan (default-prepare-plan) @ > openejb-corba-deployer --- > [INFO] Generated: > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:verify-no-dependency-change > (default-verify-no-dependency-change) @ openejb-corba-deployer --- > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-metadata > (default-prepare-metadata) @ openejb-corba-deployer --- > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:package (default-package) @ > openejb-corba-deployer --- > [INFO] Packaging module configuration: > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Enabling SLF4J API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Enabling Jakarta Commons Logging API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Enabling Log4J API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Enabling Avalon Logger API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Enabling JULI Logger API support. > log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a > "org.apache.log4j.Appender" variable. > log4j:ERROR The class "org.apache.log4j.Appender" was loaded by > log4j:ERROR [25.0] whereas object of type > log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by > [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, > parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. > log4j:ERROR Could not instantiate appender named "CONSOLE". > log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a > "org.apache.log4j.Appender" variable. > log4j:ERROR The class "org.apache.log4j.Appender" was loaded by > log4j:ERROR [25.0] whereas object of type > log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by > [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, > parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. > log4j:ERROR Could not instantiate appender named "A1". > log4j:WARN No appenders could be found for logger > (org.apache.geronimo.specs.geronimo-osgi-registry). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a > "org.apache.log4j.Appender" variable. > log4j:ERROR The class "org.apache.log4j.Appender" was loaded by > log4j:ERROR [25.0] whereas object of type > log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by > [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, > parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. > log4j:ERROR Could not instantiate appender named "A1". > > !!!!! always a significant pause here !!!!! > !!!!! from a few minutes to several minutes !!!!! > > [INFO] Started deployer: > org.apache.geronimo.framework/geronimo-gbean-deployer/3.0-SNAPSHOT/car > > !!!!! usually a significant pause here !!!!! > !!!!! from less than a minute to a few minutes !!!!! > > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:archive-car (default-archive-car) @ > openejb-corba-deployer --- > > !!!!! seemingly always a significant pause here !!!!! > !!!!! from a few minutes to 5-to-10 minutes !!!!! > > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Disabling SLF4J API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Disabling Jakarta Commons Logging API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Disabling Log4J API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Disabling Avalon Logger API support. > org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] > : Disabling JULI Logger API support. > [INFO] Expanding: > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car > into /tmp/archived-file-set.516894452.tmp > [INFO] Building jar: > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car > [INFO] Building jar: > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car > [INFO] > [INFO] --- geronimo-osgi-plugin:3.0-SNAPSHOT:verify-manifest > (verify-manifest) @ > openejb-corba-deployer --- > [INFO] > [INFO] --- ianal-maven-plugin:1.0-alpha-1:verify-legal-files (default) @ > openejb-corba-deployer --- > [INFO] Checking legal files in: openejb-corba-deployer-3.0-SNAPSHOT.car > [INFO] > [INFO] --- maven-install-plugin:2.2:install (default-install) @ > openejb-corba-deployer --- > [INFO] Installing > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car > to > /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car > [INFO] Installing > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/pom.xml > to > /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.pom > [INFO] Installing > /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/resources/META-INF/geronimo-plugin.xml > to > /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.plugin-metadata > [INFO] > [INFO] --- car-maven-plugin:3.0-SNAPSHOT:update-pluginlist > (default-update-pluginlist) @ openejb-corba-deployer --- > [INFO] > [INFO] > ------------------------------------------------------------------------ > [INFO] Building Geronimo Plugins, Corba :: Client Yoko 3.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > ... etc ... > - > >> >>> >>> Thanks for the excellent tips! >>> I'll try some things out, and look at how to reflect the tips in the wiki >>> without degrading any affirmation of Best Practices (i.e. running the test >>> cases >>> is good). >> >> No problem. Many thanks for digging into this. Definitely an area we tend to >> overlook... >> >> --kevan >>
