On 06/22/2011 01:54 PM, Kevan Miller wrote:
>
> On Jun 22, 2011, at 1:27 PM, Russell E Glaue wrote:
>
>> On 06/22/2011 12:02 PM, Kevan Miller wrote:
>>>
>>> On Jun 22, 2011, at 12:09 PM, Russell E Glaue wrote:
>
> <snip>
>
>>>
>>>>
>>>> I am using a very basic build procedure.
>>>> 1. checkout G3 trunk
>>>> 2. JAVA_HOME="/path/to/jdk/1.6.0_22"
>>>> 3. download and install a fresh maven 2.2.1
>>>> 4. MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"
>>>
>>> My settings are similar. I don't specify '-XX:ReservedCodeCacheSize=64m'.
>>>
>>> I run with the following. I know there are others running with larger max
>>> heaps. I haven't :
>>>
>>> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"
Without the "-XX:ReservedCodeCacheSize=64m" option in MAVEN_OPTS I get this
OutOfMemory error:
-
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Build Support :: Plugin 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
...<snip>...
[INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @
buildsupport-maven-plugin ---
[INFO] Compiling 3 source files to
/data/geronimo/asf-geronimo-server-trunk/framework/buildsupport/buildsupport-maven-plugin/target/classes
[INFO]
[INFO] --- gmaven-plugin:1.2:compile (default) @ buildsupport-maven-plugin ---
---------------------------------------------------
constituent[0]: file:/usr/local/apache-maven-3.0.3/lib/maven-embedder-3.0.3.jar
...<snip>...
constituent[30]: file:/usr/local/apache-maven-3.0.3/lib/maven-compat-3.0.3.jar
---------------------------------------------------
Exception in thread "main" java.lang.VirtualMachineError: out of space in
CodeCache for adapters
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:212)
at
org.codehaus.groovy.ast.builder.AstBuilderTransformation.visit(AstBuilderTransformation.groovy:62)
at
org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:268)
at
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:799)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:443)
at
org.codehaus.gmaven.runtime.v1_6.ClassCompilerFeature$ClassCompilerImpl.compile(ClassCompilerFeature.java:159)
at
org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.compile(AbstractCompileMojo.java:200)
at
org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.process(AbstractCompileMojo.java:164)
at
org.codehaus.gmaven.plugin.ComponentMojoSupport.doExecute(ComponentMojoSupport.java:60)
at org.codehaus.gmaven.plugin.MojoSupport.execute(MojoSupport.java:69)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
-
With the "-XX:ReservedCodeCacheSize=64m" flag, I do not get this error.
>>
>> The server I am build on has 1024MB of memory total, and has a dual dual-core
>> Intel(R) XEON(TM) CPU 2.00GHz with 512KB cache.
>>
>> I have tried many JVM configurations, and the one like mine and yours is the
>> best one I have found. For me, a JVM configuration with less than 400m for
>> MaxPermSize always ended with an OutOfMemory error on PermGen space.
>>
>> The Geronimo dev wiki suggests 200m for MaxPermSize, which never worked for
>> me
>> for trunk. I always assumed it was because of all the snapshot downloading
>> the
>> build process has to do. Even after the 2nd or 3rd build, 200m was never
>> enough.
>
> Right. So, MaxPermSize=200m is definitely out-of-date. There is/may be some
> bugs that are increasing this value. However, not really worth our while to
> pursue at the moment. I don't think MaxPermSize=500m is too prohibitive. More
> important things to do, ATM.
>
> That said -- I'm building on a machine with 8 gigs of real memory. So, 1024MB
> of real memory seems very small to me. And I suspect that this is the source
> of your problem... A max heap of 1024MB and MaxPermSize of 512m means your
> system may be spending a lot of time swapping virtual memory.
So wrote what are your JVM Options set at:
MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError"
Mine are:
MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m"
The machine I am using has 1024MB of real memory and is dedicated to building
geronimo (and it is not a virtual server). I set the heap max to 968m to keep
the memory usage from peaking at 100% of system capacity.
But there is a chance you may be right about the swaping. the machine runs at
100% real memory usage (expected) and about ~50% swap.
I thinking recommending a minimum amount of real memory a machine should have to
adequately compile G3 in a reasonable amount of time should be suggested.
What would you say the minimum recommendation should be? For Windows and Linux?
2GB for Windows and 1.5GB for Linux, assuming the machine is dedicated to
compiling?
>
>>
>>>
>>>> 5. mvn clean install
>>>> 6. 13 to 39 hours later == BUILD SUCCESS
>>>>
>>>>
>>>> From observing the build process, it would seem the most used (seemingly
>>>> paused)
>>>> time is right before the build process moves on to the next plugin or
>>>> assembly.
>>>>
>>>> This "seemingly paused time" after the assembly I assume is the packaging
>>>> process, because the resulting output is a huge collection.
>>>>
>>>> [INFO] Geronimo Assemblies ............................... SUCCESS [3.995s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS
>>>> [8:15.210s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS
>>>> [1:57:17.311s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS
>>>> [1:02:47.374s]
>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS
>>>> [8:08.838s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS
>>>> [1:59:16.619s]
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS
>>>> [1:08:13.854s]
>>>>
>>>> That is ~1.5 hours for the "Java EE 6 Web Profile + Jetty8" and "Java EE 6
>>>> Web
>>>> Profile + Tomcat7" assemblies.
>>>
>>> Here are my build times. This is running without tests. Which would affect
>>> the overall build time. Assembly times would be unaffected -- we don't run
>>> any tests when building assemblies.
>>>
>>> [INFO] Geronimo Assemblies ............................... SUCCESS [0.163s]
>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS [42.253s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS
>>> [1:04.295s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS
>>> [1:33.097s]
>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS [41.587s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS
>>> [1:14.409s]
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS
>>> [1:46.333s]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 28:15.199s
>>> [INFO] Finished at: Wed Jun 22 12:38:10 EDT 2011
>>> [INFO] Final Memory: 633M/1011M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>
>> Yours is about 1/3 faster than mine.
>
> I think you've missed the metric. My times are an order of magnitude (hours
> vs minutes) better than yours.
>
>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS
>>>> [1:08:13.854s]
>
> and
>
>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS
>>> [1:46.333s]
>
> That's 68 minutes vs 1 minute 46 seconds. My *total* build time was 28
> minutes.
Oh. ya. Let me rub my eyes and take another look there.... your right!
>
>> But you might have a faster processor too.
>> Obviously, this part of the build is not the time consumer for me. It is
>> nice to
>> know this part of the build is supposed to take several hours - that helps me
>> narrow down the issues.
Okay - this was an incorrect statement by me.
>>
>>>
>>>
>>>>
>>>>
>>>> For the plugins, I would assume the same, except that I notice that, of
>>>> course,
>>>> the latest snapshot dependencies are being downloaded, and the poms in the
>>>> maven
>>>> SNAPSHOT repositories are continually being accessed.
>>>>
>>>> I realize that because trunk (G3) largely depends on 3rd-party artifact
>>>> SNAPSHOTS that these repositories are checked during every build, and new
>>>> SNAPSHOT libraries are always being downloaded (as is the nature of
>>>> snapshot
>>>> releases).
>>>
>>> Correct. A local maven proxy (e.g. Nexus) can reduce the overhead.
>>
>> So it is worth noting as it already is now.
>>
>>>
>>>>
>>>> As it is not a simple process for a user to setup a maven repo proxy, and
>>>> could
>>>> be a deterrent to someone getting started if it were necessary to install a
>>>> proxy, I am looking at other methods and tips that might increase the
>>>> speed of
>>>> building G3.
>>>
>>> I've tried running with a local maven proxy. However, I've found it to be
>>> too much trouble to maintain/configure.
>>
>> Agreed. At least it is too much trouble if you are not continuously building
>> the
>> entire trunk every day on the larger scale. So too much to ask a G3 beginner
>> to do.
>
> Too much to ask of a new maven user -- I agree.
>
>>
>>>
>>>>
>>>> Is this feasible? Or would we just say, that because of the nature of
>>>> SNAPSHOT
>>>> builds and their dependencies on 3rd-party SNAPSHOT artifacts, there is no
>>>> real
>>>> means to improving build speed unless a proxy is utilized?
>>>> If the later is true, I would like to add this note in the wiki's build
>>>> documentation. And also warn those who choose to build without a proxy may
>>>> experience a long build time.
>>>
>>> While I'm thinking about it -- what is your network environment? The focus
>>> on proxies imply that your slow build times are predominantly network
>>> issue. Are you certain that is the case?
>>
>> We have about ~14Mb/s speed to the Internet. Our backplane is Gigabit
>> Ethernet.
>> We are connected to a major Internet backbone provider.
>> I have been wondering if the problem is responsiveness of the maven repo
>> sites.
>> I have been seeing on occasion a refusal to download an artifact from the
>> site,
>> where as I was able to download it within a browser from the same URL.
>> So I wonder if the repo sites are either metering the usage on a per user or
>> session basis.
>
> I'm building on my home broadband connection. Probably similar speeds. I
> think we should identify the source of your extremely slow build time.
>
> If remote repository lookups are the cause of your problems, 'mvn -o clean
> install' should result in a much faster build time. However, I suspect that
> it won't make much difference at all...
It seems a lot of "pauses" in the build process (while using the
'-DskipTests=true' flag) is at the end of building each plugin, and that looks
like this (example plugin build output)...
(Note: almost every plugin gives the log4j errors)
-
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Plugins, Corba :: Deployer 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.3:clean (default-clean) @
openejb-corba-deployer ---
[INFO]
[INFO] --- genesis-maven-plugin:2.0:validate-configuration (default) @
openejb-corba-deployer ---
[INFO]
[INFO] --- geronimo-property-plugin:3.0-SNAPSHOT:set-property (set-property) @
openejb-corba-deployer ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (default) @
openejb-corba-deployer ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.0:process (default) @
openejb-corba-deployer ---
!!!!! somewhat of a pause here, can be up to 60 seconds sometimes !!!!!
[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
valid reference.
[WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference :
template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is not a
valid reference.
[INFO]
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @
openejb-corba-deployer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/resources
[INFO] skip non existing resourceDirectory
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/filtered-resources
[INFO] Copying 3 resources
!!!!! for the plugins that use the maven-compiler-plugin:2.0.2:compile, of
course this will take time as it is compiling !!!!!
!!!!! However, the CPU usage only rises for like the first quarter of this wait
time. !!!!!
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:validate-configuration
(default-validate-configuration) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-plan (default-prepare-plan) @
openejb-corba-deployer ---
[INFO] Generated:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:verify-no-dependency-change
(default-verify-no-dependency-change) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-metadata
(default-prepare-metadata) @ openejb-corba-deployer ---
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:package (default-package) @
openejb-corba-deployer ---
[INFO] Packaging module configuration:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling SLF4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Jakarta Commons Logging API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Log4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Avalon Logger API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling JULI Logger API support.
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "CONSOLE".
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "A1".
log4j:WARN No appenders could be found for logger
(org.apache.geronimo.specs.geronimo-osgi-registry).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to a
"org.apache.log4j.Appender" variable.
log4j:ERROR The class "org.apache.log4j.Appender" was loaded by
log4j:ERROR [25.0] whereas object of type
log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by
[ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340,
parent: sun.misc.Launcher$AppClassLoader@11b86e7]].
log4j:ERROR Could not instantiate appender named "A1".
!!!!! always a significant pause here !!!!!
!!!!! from a few minutes to several minutes !!!!!
[INFO] Started deployer:
org.apache.geronimo.framework/geronimo-gbean-deployer/3.0-SNAPSHOT/car
!!!!! usually a significant pause here !!!!!
!!!!! from less than a minute to a few minutes !!!!!
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:archive-car (default-archive-car) @
openejb-corba-deployer ---
!!!!! seemingly always a significant pause here !!!!!
!!!!! from a few minutes to 5-to-10 minutes !!!!!
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling SLF4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Jakarta Commons Logging API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Log4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling Avalon Logger API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Disabling JULI Logger API support.
[INFO] Expanding:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
into /tmp/archived-file-set.516894452.tmp
[INFO] Building jar:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO] Building jar:
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO]
[INFO] --- geronimo-osgi-plugin:3.0-SNAPSHOT:verify-manifest (verify-manifest) @
openejb-corba-deployer ---
[INFO]
[INFO] --- ianal-maven-plugin:1.0-alpha-1:verify-legal-files (default) @
openejb-corba-deployer ---
[INFO] Checking legal files in: openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO]
[INFO] --- maven-install-plugin:2.2:install (default-install) @
openejb-corba-deployer ---
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/pom.xml
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.pom
[INFO] Installing
/data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/resources/META-INF/geronimo-plugin.xml
to
/home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.plugin-metadata
[INFO]
[INFO] --- car-maven-plugin:3.0-SNAPSHOT:update-pluginlist
(default-update-pluginlist) @ openejb-corba-deployer ---
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Geronimo Plugins, Corba :: Client Yoko 3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
... etc ...
-
>
>>
>> Thanks for the excellent tips!
>> I'll try some things out, and look at how to reflect the tips in the wiki
>> without degrading any affirmation of Best Practices (i.e. running the test
>> cases
>> is good).
>
> No problem. Many thanks for digging into this. Definitely an area we tend to
> overlook...
>
> --kevan
>
