On 06/22/2011 04:49 PM, David Jencks wrote: > I've had disk thrashing up my build times from ~30 min to ~2 hours, but never > this much. I have no evidence but would suspect a DNS problem resulting in > multiple DNS timeouts for each module. > > Do you have a more-memory-endowed machine on this network you could try a > build on to see if it's faster? > > thanks > david jencks
I got another server in our lab, and put 2GB of memory in it. It has a single 2.4GHz dual-core CPU. I'll run a build on it, and we can compare. -RG > > On Jun 22, 2011, at 1:32 PM, Russell E Glaue wrote: > >> On 06/22/2011 01:54 PM, Kevan Miller wrote: >>> >>> On Jun 22, 2011, at 1:27 PM, Russell E Glaue wrote: >>> >>>> On 06/22/2011 12:02 PM, Kevan Miller wrote: >>>>> >>>>> On Jun 22, 2011, at 12:09 PM, Russell E Glaue wrote: >>> >>> <snip> >>> >>>>> >>>>>> >>>>>> I am using a very basic build procedure. >>>>>> 1. checkout G3 trunk >>>>>> 2. JAVA_HOME="/path/to/jdk/1.6.0_22" >>>>>> 3. download and install a fresh maven 2.2.1 >>>>>> 4. MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m >>>>>> -XX:ReservedCodeCacheSize=64m" >>>>> >>>>> My settings are similar. I don't specify '-XX:ReservedCodeCacheSize=64m'. >>>>> >>>>> I run with the following. I know there are others running with larger max >>>>> heaps. I haven't : >>>>> >>>>> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m >>>>> -XX:+HeapDumpOnOutOfMemoryError" >> >> Without the "-XX:ReservedCodeCacheSize=64m" option in MAVEN_OPTS I get this >> OutOfMemory error: >> - >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Building Geronimo Build Support :: Plugin 3.0-SNAPSHOT >> [INFO] >> ------------------------------------------------------------------------ >> ...<snip>... >> [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ >> buildsupport-maven-plugin --- >> [INFO] Compiling 3 source files to >> /data/geronimo/asf-geronimo-server-trunk/framework/buildsupport/buildsupport-maven-plugin/target/classes >> [INFO] >> [INFO] --- gmaven-plugin:1.2:compile (default) @ buildsupport-maven-plugin >> --- >> --------------------------------------------------- >> constituent[0]: >> file:/usr/local/apache-maven-3.0.3/lib/maven-embedder-3.0.3.jar >> ...<snip>... >> constituent[30]: >> file:/usr/local/apache-maven-3.0.3/lib/maven-compat-3.0.3.jar >> --------------------------------------------------- >> Exception in thread "main" java.lang.VirtualMachineError: out of space in >> CodeCache for adapters >> at >> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:212) >> at >> org.codehaus.groovy.ast.builder.AstBuilderTransformation.visit(AstBuilderTransformation.groovy:62) >> at >> org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:268) >> at >> org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:799) >> at >> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:464) >> at >> org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:443) >> at >> org.codehaus.gmaven.runtime.v1_6.ClassCompilerFeature$ClassCompilerImpl.compile(ClassCompilerFeature.java:159) >> at >> org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.compile(AbstractCompileMojo.java:200) >> at >> org.codehaus.gmaven.plugin.compile.AbstractCompileMojo.process(AbstractCompileMojo.java:164) >> at >> org.codehaus.gmaven.plugin.ComponentMojoSupport.doExecute(ComponentMojoSupport.java:60) >> at org.codehaus.gmaven.plugin.MojoSupport.execute(MojoSupport.java:69) >> at >> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) >> at >> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) >> at >> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) >> - >> >> With the "-XX:ReservedCodeCacheSize=64m" flag, I do not get this error. >> >> >>>> >>>> The server I am build on has 1024MB of memory total, and has a dual >>>> dual-core >>>> Intel(R) XEON(TM) CPU 2.00GHz with 512KB cache. >>>> >>>> I have tried many JVM configurations, and the one like mine and yours is >>>> the >>>> best one I have found. For me, a JVM configuration with less than 400m for >>>> MaxPermSize always ended with an OutOfMemory error on PermGen space. >>>> >>>> The Geronimo dev wiki suggests 200m for MaxPermSize, which never worked >>>> for me >>>> for trunk. I always assumed it was because of all the snapshot downloading >>>> the >>>> build process has to do. Even after the 2nd or 3rd build, 200m was never >>>> enough. >>> >>> Right. So, MaxPermSize=200m is definitely out-of-date. There is/may be some >>> bugs that are increasing this value. However, not really worth our while to >>> pursue at the moment. I don't think MaxPermSize=500m is too prohibitive. >>> More important things to do, ATM. >>> >>> That said -- I'm building on a machine with 8 gigs of real memory. So, >>> 1024MB of real memory seems very small to me. And I suspect that this is >>> the source of your problem... A max heap of 1024MB and MaxPermSize of 512m >>> means your system may be spending a lot of time swapping virtual memory. >> >> So wrote what are your JVM Options set at: >> MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError" >> Mine are: >> MAVEN_OPTS="-Xmx968m -XX:MaxPermSize=500m -XX:ReservedCodeCacheSize=64m" >> >> The machine I am using has 1024MB of real memory and is dedicated to building >> geronimo (and it is not a virtual server). I set the heap max to 968m to keep >> the memory usage from peaking at 100% of system capacity. >> >> But there is a chance you may be right about the swaping. the machine runs at >> 100% real memory usage (expected) and about ~50% swap. >> >> >> I thinking recommending a minimum amount of real memory a machine should >> have to >> adequately compile G3 in a reasonable amount of time should be suggested. >> What would you say the minimum recommendation should be? For Windows and >> Linux? >> 2GB for Windows and 1.5GB for Linux, assuming the machine is dedicated to >> compiling? >> >>> >>>> >>>>> >>>>>> 5. mvn clean install >>>>>> 6. 13 to 39 hours later == BUILD SUCCESS >>>>>> >>>>>> >>>>>> From observing the build process, it would seem the most used (seemingly >>>>>> paused) >>>>>> time is right before the build process moves on to the next plugin or >>>>>> assembly. >>>>>> >>>>>> This "seemingly paused time" after the assembly I assume is the packaging >>>>>> process, because the resulting output is a huge collection. >>>>>> >>>>>> [INFO] Geronimo Assemblies ............................... SUCCESS >>>>>> [3.995s] >>>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS >>>>>> [8:15.210s] >>>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS >>>>>> [1:57:17.311s] >>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS >>>>>> [1:02:47.374s] >>>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS >>>>>> [8:08.838s] >>>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS >>>>>> [1:59:16.619s] >>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>>> [1:08:13.854s] >>>>>> >>>>>> That is ~1.5 hours for the "Java EE 6 Web Profile + Jetty8" and "Java EE >>>>>> 6 Web >>>>>> Profile + Tomcat7" assemblies. >>>>> >>>>> Here are my build times. This is running without tests. Which would >>>>> affect the overall build time. Assembly times would be unaffected -- we >>>>> don't run any tests when building assemblies. >>>>> >>>>> [INFO] Geronimo Assemblies ............................... SUCCESS >>>>> [0.163s] >>>>> [INFO] Geronimo Assemblies :: Minimal + Jetty8 ........... SUCCESS >>>>> [42.253s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Jetty8 SUCCESS >>>>> [1:04.295s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Jetty8 ......... SUCCESS >>>>> [1:33.097s] >>>>> [INFO] Geronimo Assemblies :: Minimal + Tomcat7 .......... SUCCESS >>>>> [41.587s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 Web Profile + Tomcat7 SUCCESS >>>>> [1:14.409s] >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>> [1:46.333s] >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> [INFO] BUILD SUCCESS >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> [INFO] Total time: 28:15.199s >>>>> [INFO] Finished at: Wed Jun 22 12:38:10 EDT 2011 >>>>> [INFO] Final Memory: 633M/1011M >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>> >>>> Yours is about 1/3 faster than mine. >>> >>> I think you've missed the metric. My times are an order of magnitude (hours >>> vs minutes) better than yours. >>> >>>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>>> [1:08:13.854s] >>> >>> and >>> >>>>> [INFO] Geronimo Assemblies :: Java EE 6 + Tomcat7 ........ SUCCESS >>>>> [1:46.333s] >>> >>> That's 68 minutes vs 1 minute 46 seconds. My *total* build time was 28 >>> minutes. >> >> Oh. ya. Let me rub my eyes and take another look there.... your right! >> >>> >>>> But you might have a faster processor too. >>>> Obviously, this part of the build is not the time consumer for me. It is >>>> nice to >>>> know this part of the build is supposed to take several hours - that helps >>>> me >>>> narrow down the issues. >> >> Okay - this was an incorrect statement by me. >> >>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> For the plugins, I would assume the same, except that I notice that, of >>>>>> course, >>>>>> the latest snapshot dependencies are being downloaded, and the poms in >>>>>> the maven >>>>>> SNAPSHOT repositories are continually being accessed. >>>>>> >>>>>> I realize that because trunk (G3) largely depends on 3rd-party artifact >>>>>> SNAPSHOTS that these repositories are checked during every build, and new >>>>>> SNAPSHOT libraries are always being downloaded (as is the nature of >>>>>> snapshot >>>>>> releases). >>>>> >>>>> Correct. A local maven proxy (e.g. Nexus) can reduce the overhead. >>>> >>>> So it is worth noting as it already is now. >>>> >>>>> >>>>>> >>>>>> As it is not a simple process for a user to setup a maven repo proxy, >>>>>> and could >>>>>> be a deterrent to someone getting started if it were necessary to >>>>>> install a >>>>>> proxy, I am looking at other methods and tips that might increase the >>>>>> speed of >>>>>> building G3. >>>>> >>>>> I've tried running with a local maven proxy. However, I've found it to be >>>>> too much trouble to maintain/configure. >>>> >>>> Agreed. At least it is too much trouble if you are not continuously >>>> building the >>>> entire trunk every day on the larger scale. So too much to ask a G3 >>>> beginner to do. >>> >>> Too much to ask of a new maven user -- I agree. >>> >>>> >>>>> >>>>>> >>>>>> Is this feasible? Or would we just say, that because of the nature of >>>>>> SNAPSHOT >>>>>> builds and their dependencies on 3rd-party SNAPSHOT artifacts, there is >>>>>> no real >>>>>> means to improving build speed unless a proxy is utilized? >>>>>> If the later is true, I would like to add this note in the wiki's build >>>>>> documentation. And also warn those who choose to build without a proxy >>>>>> may >>>>>> experience a long build time. >>>>> >>>>> While I'm thinking about it -- what is your network environment? The >>>>> focus on proxies imply that your slow build times are predominantly >>>>> network issue. Are you certain that is the case? >>>> >>>> We have about ~14Mb/s speed to the Internet. Our backplane is Gigabit >>>> Ethernet. >>>> We are connected to a major Internet backbone provider. >>>> I have been wondering if the problem is responsiveness of the maven repo >>>> sites. >>>> I have been seeing on occasion a refusal to download an artifact from the >>>> site, >>>> where as I was able to download it within a browser from the same URL. >>>> So I wonder if the repo sites are either metering the usage on a per user >>>> or >>>> session basis. >>> >>> I'm building on my home broadband connection. Probably similar speeds. I >>> think we should identify the source of your extremely slow build time. >>> >>> If remote repository lookups are the cause of your problems, 'mvn -o clean >>> install' should result in a much faster build time. However, I suspect that >>> it won't make much difference at all... >> >> It seems a lot of "pauses" in the build process (while using the >> '-DskipTests=true' flag) is at the end of building each plugin, and that >> looks >> like this (example plugin build output)... >> (Note: almost every plugin gives the log4j errors) >> - >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Building Geronimo Plugins, Corba :: Deployer 3.0-SNAPSHOT >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] >> [INFO] --- maven-clean-plugin:2.3:clean (default-clean) @ >> openejb-corba-deployer --- >> [INFO] >> [INFO] --- genesis-maven-plugin:2.0:validate-configuration (default) @ >> openejb-corba-deployer --- >> [INFO] >> [INFO] --- geronimo-property-plugin:3.0-SNAPSHOT:set-property (set-property) >> @ >> openejb-corba-deployer --- >> [INFO] >> [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (default) @ >> openejb-corba-deployer --- >> [INFO] >> [INFO] --- maven-remote-resources-plugin:1.0:process (default) @ >> openejb-corba-deployer --- >> >> !!!!! somewhat of a pause here, can be up to 60 seconds sometimes !!!!! >> >> [WARNING] org.apache.velocity.runtime.exception.ReferenceException: >> reference : >> template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is >> not a >> valid reference. >> [WARNING] org.apache.velocity.runtime.exception.ReferenceException: >> reference : >> template = META-INF/DEPENDENCIES.vm [line 40,column 14] : $license.name is >> not a >> valid reference. >> [INFO] >> [INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ >> openejb-corba-deployer --- >> [INFO] Using 'UTF-8' encoding to copy filtered resources. >> [INFO] skip non existing resourceDirectory >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/resources >> [INFO] skip non existing resourceDirectory >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/src/main/filtered-resources >> [INFO] Copying 3 resources >> >> !!!!! for the plugins that use the maven-compiler-plugin:2.0.2:compile, of >> course this will take time as it is compiling !!!!! >> !!!!! However, the CPU usage only rises for like the first quarter of this >> wait >> time. !!!!! >> >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:validate-configuration >> (default-validate-configuration) @ openejb-corba-deployer --- >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-plan (default-prepare-plan) >> @ >> openejb-corba-deployer --- >> [INFO] Generated: >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:verify-no-dependency-change >> (default-verify-no-dependency-change) @ openejb-corba-deployer --- >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:prepare-metadata >> (default-prepare-metadata) @ openejb-corba-deployer --- >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:package (default-package) @ >> openejb-corba-deployer --- >> [INFO] Packaging module configuration: >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/work/plan.xml >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Enabling SLF4J API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Enabling Jakarta Commons Logging API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Enabling Log4J API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Enabling Avalon Logger API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Enabling JULI Logger API support. >> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to >> a >> "org.apache.log4j.Appender" variable. >> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by >> log4j:ERROR [25.0] whereas object of type >> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by >> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, >> parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. >> log4j:ERROR Could not instantiate appender named "CONSOLE". >> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to >> a >> "org.apache.log4j.Appender" variable. >> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by >> log4j:ERROR [25.0] whereas object of type >> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by >> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, >> parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. >> log4j:ERROR Could not instantiate appender named "A1". >> log4j:WARN No appenders could be found for logger >> (org.apache.geronimo.specs.geronimo-osgi-registry). >> log4j:WARN Please initialize the log4j system properly. >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for >> more info. >> log4j:ERROR A "org.apache.log4j.ConsoleAppender" object is not assignable to >> a >> "org.apache.log4j.Appender" variable. >> log4j:ERROR The class "org.apache.log4j.Appender" was loaded by >> log4j:ERROR [25.0] whereas object of type >> log4j:ERROR "org.apache.log4j.ConsoleAppender" was loaded by >> [ClassRealm[plugin>org.apache.geronimo.buildsupport:car-maven-plugin:3.0-SNAPSHOT--1708152340, >> parent: sun.misc.Launcher$AppClassLoader@11b86e7]]. >> log4j:ERROR Could not instantiate appender named "A1". >> >> !!!!! always a significant pause here !!!!! >> !!!!! from a few minutes to several minutes !!!!! >> >> [INFO] Started deployer: >> org.apache.geronimo.framework/geronimo-gbean-deployer/3.0-SNAPSHOT/car >> >> !!!!! usually a significant pause here !!!!! >> !!!!! from less than a minute to a few minutes !!!!! >> >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:archive-car (default-archive-car) @ >> openejb-corba-deployer --- >> >> !!!!! seemingly always a significant pause here !!!!! >> !!!!! from a few minutes to 5-to-10 minutes !!!!! >> >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Disabling SLF4J API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Disabling Jakarta Commons Logging API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Disabling Log4J API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Disabling Avalon Logger API support. >> org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] >> : Disabling JULI Logger API support. >> [INFO] Expanding: >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car >> into /tmp/archived-file-set.516894452.tmp >> [INFO] Building jar: >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car >> [INFO] Building jar: >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car >> [INFO] >> [INFO] --- geronimo-osgi-plugin:3.0-SNAPSHOT:verify-manifest >> (verify-manifest) @ >> openejb-corba-deployer --- >> [INFO] >> [INFO] --- ianal-maven-plugin:1.0-alpha-1:verify-legal-files (default) @ >> openejb-corba-deployer --- >> [INFO] Checking legal files in: openejb-corba-deployer-3.0-SNAPSHOT.car >> [INFO] >> [INFO] --- maven-install-plugin:2.2:install (default-install) @ >> openejb-corba-deployer --- >> [INFO] Installing >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/openejb-corba-deployer-3.0-SNAPSHOT.car >> to >> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.car >> [INFO] Installing >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/pom.xml >> to >> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.pom >> [INFO] Installing >> /data/geronimo/asf-geronimo-server-trunk/plugins/corba/openejb-corba-deployer/target/resources/META-INF/geronimo-plugin.xml >> to >> /home/ger/.m2/repository/org/apache/geronimo/configs/openejb-corba-deployer/3.0-SNAPSHOT/openejb-corba-deployer-3.0-SNAPSHOT.plugin-metadata >> [INFO] >> [INFO] --- car-maven-plugin:3.0-SNAPSHOT:update-pluginlist >> (default-update-pluginlist) @ openejb-corba-deployer --- >> [INFO] >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Building Geronimo Plugins, Corba :: Client Yoko 3.0-SNAPSHOT >> [INFO] >> ------------------------------------------------------------------------ >> ... etc ... >> - >> >>> >>>> >>>> Thanks for the excellent tips! >>>> I'll try some things out, and look at how to reflect the tips in the wiki >>>> without degrading any affirmation of Best Practices (i.e. running the test >>>> cases >>>> is good). >>> >>> No problem. Many thanks for digging into this. Definitely an area we tend >>> to overlook... >>> >>> --kevan >>> >
