The current version of m2e runs mvn embeded inside the eclipse jvm. Because eclipse has one classloader isolation system, and maven has another, the sum total is a wildly effective recipe for running out of VM.
This is exacerbated by projects like CXF, as Ron mentions, that fire up the JDK or other big consumers of permgen space. I'm afraid that M2E + a very large project is a recipe for a 64-bit laptop and >1G of JVM memory, with or without version ranges. None of which takes away the value of some profiling to see if there's some point defect in maven which is effectively leaking vm. On Tue, Jun 21, 2011 at 11:20 AM, Ron Wheeler <rwhee...@artifact-software.com> wrote: > On 21/06/2011 10:55 AM, Paul French wrote: >> >> Hello Ron, >> >> thanks for your comments. See inline comments. >> >> Cheers >> >> On 21/06/2011 15:38, Ron Wheeler wrote: >>> >>> On 21/06/2011 9:45 AM, Paul French wrote: >>>> >>>> On 21/06/2011 14:25, Ron Wheeler wrote: >>>>> >>>>> 500Mb is not a lot of memory for a Java program. >>>> >>>> It is when all I can give eclipse is 900MB. Beyond that the JVM fails to >>>> start. M2E as far as I know does not start a new JVM when building. >>> >>> Perhaps someone else can settle this. >>> >>> http://maven.40175.n5.nabble.com/setting-java-memory-options-for-m2-plugin-td138828.html >>> seems to indicate that Maven runs in its own JVM with its own memory >>> configuration. >> >> I hope you are right. But that particular post is for launching maven >> externally to eclipse. The m2e plugin itself uses an embedded maven for >> dependency management. It is the dependency management that is failing with >> out of memory. > > That is something that I have never seen but we have broken our portal > application into over 70 separately-built modules and have built aggregation > projects to create libraries of third party dependencies. For example, we > have a single jar for Spring, Hibernate and MySQL since almost every one of > our projects need these. This is referenced as single "provided" dependency > since we only install 1 copy in the Tomcat shared library. > When combined with proper exclusions in the creation of the library (no need > for 70 copies of commons-logging) , our dependency graphs are very small and > our builds very fast even if the module requires 70 or more third party > libraries to compile and run. > > >> >>>> >>>>> We have noticed that some libraries such as Apache's CXF webservices >>>>> will require additional memory to be added to Maven if you are building >>>>> applications that include it. >>>>> Since virtual memory is virtually free, you might as well just accept >>>>> reality and increase the VM. >>>>> >>>>> I can not imagine anyone doing software development with a modern IDE >>>>> that would have trouble finding 1Gb of VM. >>>> >>>> On my 2GB laptop all I can give eclipse is 900MB >>> >>> You are mixing real and virtual. >>> You can give it as much virtual as you need and it will find the real >>> memory (usually a lot less) that it can use. >>> You should have your Windows VM set to at least 4Gb on a 2Gb RAM laptop >>> doing software development. >> >> I'm not mixing real and virtual. I was just indicating the maximum amount >> of memory I could give eclipse on my laptop. I don't care whether it is real >> or virtual I let windows handle that. >> >> If someone can tell me how I can give eclipse more then a 1G of memory >> (windows XP) then that would be great. At the moment I use a shortcut: >> >> eclipse.exe -vmargs -Xmx900M >> >> If I set to 1000M then the JVM fails to start. >> >>> >>> All our guys have 2Gb laptops and can set virtual memory to 1 Gb for >>> Eclipse and 1 Gb for the JVMs that it spawns. >>> >>> It is only for a short time anyway and what else are you trying to do >>> while it builds? >>> You want it to use every bit of real RAM that it needs to finish quickly. >>> >>> Anyway, there is nothing that you can do to make it use less memory >>> unless you want to break your builds down and even then, you may need >>> libraries like CXF that just take a lot VM to build. >>> >>> >>> Ron >>>> >>>> >>>>> >>>>> We build under Eclipse STS but I don't think that it makes any >>>>> difference since Maven is running in its own VM. >>>>> >>>>> I would not use version ranges since I like to know what I am putting >>>>> into production. >>>>> Once you initially set the versions of what you are supporting, it is >>>>> not difficult to maintain that list as you work your way through new >>>>> versions of third party code. >>>>> We do it at the start of each of our release cycles. >>>>> We usually stick with the existing set from the last release unless >>>>> there is a reason to move. >>>>> The version configuration for a new release is setup and managed by the >>>>> entire team (small part of the release startup) and unless something >>>>> catches >>>>> fire, we stick with our set throughout the life of our release. >>>> >>>> We have 100's of bundles and usually most in flux. Someone put's a bug >>>> fix in a bundle, I do not want to have to *know*, as long as it is >>>> compatible with what I require which we automatically enforce via PDE API >>>> analysis tooling. At release time we know what is delivered and can >>>> re-produce the build since an exact version POM is built for us and >>>> packaged >>>> in the artifact with the version range one backed up. >>>> >>>> I can understand people being resistant to version ranges but with OSGi >>>> more and more jars are broken down into smaller components (and so >>>> projects) >>>> and version ranging really helps if you enforce semantic versioning between >>>> bundles. >>>> >>>> Nevertheless there is a problem with maven 3.0.3 when using version >>>> ranges. >>>> >>>>> >>>>> >>>>> Ron >>>>> >>>>> On 21/06/2011 8:59 AM, Benson Margulies wrote: >>>>>> >>>>>> Yes this came to the list. >>>>>> >>>>>> *someone* is going to have to run yourkit or jprofiler on a real >>>>>> version of your problem. >>>>>> >>>>>> Of course, the person best positioned to do that would be, ahem, you. >>>>>> Unless you could give access to, well, me. >>>>>> >>>>>> >>>>>> It would be a giant public service for there to be a sufficiently >>>>>> complex example of this sort of thing available for development and >>>>>> testing, .... and it would pay your organization off in the long run >>>>>> in terms of maven working better for you. >>>>>> >>>>>> On Tue, Jun 21, 2011 at 6:42 AM, Paul French<paul.fre...@kirona.com> >>>>>> wrote: >>>>>>> >>>>>>> Thanks for reply - see inline >>>>>>> >>>>>>> On 21/06/2011 11:27, B Smith-Mannschott wrote: >>>>>>>> >>>>>>>> On Mon, Jun 20, 2011 at 19:54, Paul French<paul.fre...@kirona.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Below is some filtered output from a maven build (showing maven >>>>>>>>> meta data >>>>>>>>> being downloaded for one artifact). All my dependencies use version >>>>>>>>> ranges >>>>>>>>> of the form [1.0.0.SNAPSHOT,2.0.0.SNAPSHOT) >>>>>>>>> >>>>>>>>> In general the build fails with out of memory. I can increase the >>>>>>>>> heap >>>>>>>>> size >>>>>>>>> and the build is successful, but a relatively simple build is now >>>>>>>>> requiring >>>>>>>>> 500MB heap size. >>>>>>>>> >>>>>>>>> First question is why is maven re-checking and downloading the same >>>>>>>>> meta >>>>>>>>> data over and over again? >>>>>>>>> >>>>>>>>> If I add additional version ranged dependencies to my pom then the >>>>>>>>> problem >>>>>>>>> seems to get exponentially worse. For a larger build so many >>>>>>>>> requests are >>>>>>>>> made so quickly to the nexus remote repository that I start to see >>>>>>>>> "Error >>>>>>>>> transferring file: Address already in use: connect". >>>>>>>>> >>>>>>>>> Any ideas how to reduce the memory usage? Do not say "do not use >>>>>>>>> version >>>>>>>>> ranges". We use semantic versioning enforced by API analysis tools >>>>>>>>> for >>>>>>>>> each >>>>>>>>> of our modules (OSGi bundles) so we require version ranges. >>>>>>>>> >>>>>>>>> If I change all version range dependencies to an exact dependency >>>>>>>>> (a >>>>>>>>> serious >>>>>>>>> amount of pom editing) then all is fine. As you can tell below we >>>>>>>>> have 3 >>>>>>>>> main internal remote repos setup (kirona, kirona-snapshot and >>>>>>>>> third.party.libraries). We do not specify version ranges on >>>>>>>>> libraries >>>>>>>>> that >>>>>>>>> are not in our control e.g. log4j etc etc >>>>>>>>> >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/third-party/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-snapshots/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (354 B at 21.6 KB/sec) >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (318 B at 6.6 KB/sec) >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-snapshots/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-knopflerfish/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/third-party/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (318 B at 10.0 KB/sec) >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-snapshots/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (354 B at 11.2 KB/sec) >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-snapshots/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/third-party/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-knopflerfish/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (318 B at 9.7 KB/sec) >>>>>>>>> Downloaded: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona-snapshots/com/kirona/server/third.party.libraries/maven-metadata.xml >>>>>>>>> (354 B at 10.8 KB/sec) >>>>>>>>> Downloading: >>>>>>>>> >>>>>>>>> >>>>>>>>> http://fuji:8081/nexus/content/repositories/kirona/com/kirona/serve........... >>>>>>>>> >>>>>>>>> The above problems renders eclipse useless with m2e, it causes >>>>>>>>> eclipse to >>>>>>>>> hang on start up (build automatically set to true) and eventually >>>>>>>>> die!! >>>>>>>>> >>>>>>>>> I'm starting to come to the conclusion that between maven and m2e >>>>>>>>> version >>>>>>>>> ranges just do not work very well. I hope I am wrong!! >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Paul >>>>>>>> >>>>>>>> In my a case I found Maven slurping metadata on every build was >>>>>>>> related to how I had configured<updatePolicy>always</updatePolicy> >>>>>>>> in >>>>>>>> repositories profile defined in my settings.xml (It had been a >>>>>>>> "temporary" change which got forgotten...) >>>>>>> >>>>>>> I did circumvent the problem as you suggest by setting my repos to >>>>>>> have an >>>>>>> updatePolicy >>>>>>> of interval:10 so during a build, maven does not re-download the same >>>>>>> maven meta data file. However not very complicated builds are still >>>>>>> failing >>>>>>> with out of memory even with 1024M of heap space. So this workaround >>>>>>> just >>>>>>> speeds up the inevitable out of memory error appearing. >>>>>>> >>>>>>>> I got burned twice by version ranges. (It's at least a few years >>>>>>>> ago, >>>>>>>> so I don't remember the specifics.) I haven't touched them since. >>>>>>>> Also, their meaning hinges on how Maven chooses to sort versions, >>>>>>>> and >>>>>>>> I've not yet seen a sufficiently unambiguous definition of that. >>>>>>>> Consider that Maven will accept aNyOld3.2Crap-X1r as a version >>>>>>>> number. >>>>>>>> Even if you stick to the 'parse-able' versions of the form >>>>>>>> 1.2.3-mumble. >>>>>>>> >>>>>>>> 1.2.3-SNAPSHOT< 1.2.3 >>>>>>>> 1.2.3-beta10< 1.2.3-beta2 >>>>>>>> 1.2.3-beta1-SNAPSHOT< 1.2.3-beta1 >>>>>>>> 1.2.3-beta1>?< 1.2.3? >>>>>>>> 1.2.3-beta1-SNAPSHOT>?< 1.2.3-SNAPSHOT >>>>>>> >>>>>>> Hence why we use ranges like [1.0.0.SNAPSHOT,2.0.0.SNAPSHOT) - i.e. >>>>>>> happily >>>>>>> depend on a 1.X version including a work in progress 1.0.0.SNAPSHOT >>>>>>> but do >>>>>>> not depend >>>>>>> on any 2.0.0 version, hence why we have excluded 2.0.0.SNAPSHOT since >>>>>>> like >>>>>>> you said 2.0.0.SNAPSHOT< 2.0.0 >>>>>>> >>>>>>>> Take your best guess. >>>>>>>> >>>>>>>> Sticking to using only SNAPSHOTs and single concrete versions is >>>>>>>> more >>>>>>>> work if you have a large number of interdependent projects. It does >>>>>>>> have the advantage of recording in your version control system very >>>>>>>> specific information about which versions go (went) together. >>>>>>>> >>>>>>> All true, but at release time you can use the maven plugin >>>>>>> versions:resolve-ranges which will backup your pom and then modify >>>>>>> with the >>>>>>> versions actually used. >>>>>>> >>>>>>>> // ben >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>>>>>>> For additional commands, e-mail: users-h...@maven.apache.org >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>>>>>> For additional commands, e-mail: users-h...@maven.apache.org >>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>>>>> For additional commands, e-mail: users-h...@maven.apache.org >>>>>> >>>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>>>> For additional commands, e-mail: users-h...@maven.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>>> For additional commands, e-mail: users-h...@maven.apache.org >>>> >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >>> For additional commands, e-mail: users-h...@maven.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org >> For additional commands, e-mail: users-h...@maven.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
