Re: [infinispan-dev] How to run the testsuite?

[Mircea Markus]

I've just run it on master and didn't get OOM. well I'm using osx. Are you 
running it on master or a particular branch? Which module crashes?
e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the reason 
it hasn't been integrated yet.

On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:

> Hi all,
> after reviewing some pull requests, I'm since a couple of days unable
> to run the testsuite; since Anna's fixes affect many modules I'm
> trying to run the testsuite of the whole project, as we should always
> do but I admit I haven't done it in a while because of the core module
> failures.
> 
> So I run:
> $ mvn -fn clean install
> 
> using -fn to have it continue after the core failures.
> 
> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> sure this was good enough some months back.
> 
> Second attempt slowed down like crazy, and I found a warning about
> having filled the code cache size, so doubled it to 200M.
> 
> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> -XX:MaxPermSize=380M which should be plenty?
> 
> This is :
> java version "1.6.0_43"
> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> 
> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> -XX:ReservedCodeCacheSize=200M
> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> 
> My custom log configuration just disables trace & debug.
> 
> Going to try now with larger PermGen and different JVMs but it looks
> quite bad.. any other suggestion?
> (I do have the security limits setup properly)
> 
> Sanne
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)





___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Sanne Grinovero]

I'm testing master, at da5c3f0

Just killed a run which was using

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

this time again an OOM (while I have 2GB !), last sign of life came
from the "Rolling Upgrade Tooling"

I'm not going to merge/review any pull request until this works.

Sanne

On 20 March 2013 12:09, Mircea Markus  wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are you 
> running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
> reason it hasn't been integrated yet.
>
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>>
>> So I run:
>> $ mvn -fn clean install
>>
>> using -fn to have it continue after the core failures.
>>
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>>
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>>
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>>
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>
>> My custom log configuration just disables trace & debug.
>>
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>>
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Tristan Tarrant]

Sanne, turn on CompressedOops ? Still those requirements are indeed 
ridiculous.

Tristan

On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus  wrote:
>> I've just run it on master and didn't get OOM. well I'm using osx. Are you 
>> running it on master or a particular branch? Which module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
>> reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should always
>>> do but I admit I haven't done it in a while because of the core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>>
>>
>>
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Dan Berindei]

The problem is that we still leak threads in almost every module, and that
means we keep a copy of the core classes (and all their dependencies) for
every module. Of course, some modules' dependencies are already oversized,
so keeping only one copy is already too much...

I admit I don't run the whole test suite too often either, but I recently
changed the Cloudbees settings to get rid of the OOM there. It uses about
550MB of permgen by the end of the test suite, without
-XX:+UseCompressedOops. These are the settings I used:

-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
-XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
-XX:MaxPermSize=700M


Cheers
Dan



On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant wrote:

> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
>
> Tristan
>
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus  wrote:
> >> I've just run it on master and didn't get OOM. well I'm using osx. Are
> you running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
> the reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days unable
> >>> to run the testsuite; since Anna's fixes affect many modules I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> ___
> >>> infinispan-dev mailing list
> >>> infinispan-dev@lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> ___
> >> infinispan-dev mailing list
> >> infinispan-dev@lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Adrian Nistor]

I've also tried changing the fork mode of surefire from 'none' to 'once' 
and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  
Previously I did not complete, 500mb was not enough.

Anyone knows why surefire was not allowed to fork?

Haven't tried to analyze closely the heap yet but first thing I noticed 
is 15% of it is occupied by 19 ComponentMetadataRepo instances, 
which probably is not the root cause of this issue, but is odd anyway :).


On 03/20/2013 05:12 PM, Dan Berindei wrote:
The problem is that we still leak threads in almost every module, and 
that means we keep a copy of the core classes (and all their 
dependencies) for every module. Of course, some modules' dependencies 
are already oversized, so keeping only one copy is already too much...


I admit I don't run the whole test suite too often either, but I 
recently changed the Cloudbees settings to get rid of the OOM there. 
It uses about 550MB of permgen by the end of the test suite, without 
-XX:+UseCompressedOops. These are the settings I used:


-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k 
-Xms100m -Xmx900m -XX:MaxPermSize=700M



Cheers
Dan



On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant > wrote:


Sanne, turn on CompressedOops ? Still those requirements are indeed
ridiculous.

Tristan

On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus mailto:mmar...@redhat.com>> wrote:
>> I've just run it on master and didn't get OOM. well I'm using
osx. Are you running it on master or a particular branch? Which
module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party -
that's the reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days
unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should
always
>>> do but I admit I haven't done it in a while because of the
core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm
pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm
running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it
looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org

>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org )
>>
>>
>>
>>
>>
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org

>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org

> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org 

Re: [infinispan-dev] How to run the testsuite?

[Mircea Markus]

On 20 Mar 2013, at 15:12, Dan Berindei wrote:

> The problem is that we still leak threads in almost every module, and that 
> means we keep a copy of the core classes (and all their dependencies) for 
> every module. Of course, some modules' dependencies are already oversized, so 
> keeping only one copy is already too much...
do we have a JIRA for this?

> 
> I admit I don't run the whole test suite too often either, but I recently 
> changed the Cloudbees settings to get rid of the OOM there. It uses about 
> 550MB of permgen by the end of the test suite, without 
> -XX:+UseCompressedOops. These are the settings I used:
> 
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
> -XX:MaxPermSize=700M

Thanks for sharing this Dan!

> 
> 
> Cheers
> Dan
> 
> 
> 
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  wrote:
> Sanne, turn on CompressedOops ? Still those requirements are indeed
> ridiculous.
> 
> Tristan
> 
> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > I'm testing master, at da5c3f0
> >
> > Just killed a run which was using
> >
> > java version "1.7.0_17"
> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >
> > this time again an OOM (while I have 2GB !), last sign of life came
> > from the "Rolling Upgrade Tooling"
> >
> > I'm not going to merge/review any pull request until this works.
> >
> > Sanne
> >
> > On 20 March 2013 12:09, Mircea Markus  wrote:
> >> I've just run it on master and didn't get OOM. well I'm using osx. Are you 
> >> running it on master or a particular branch? Which module crashes?
> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
> >> reason it hasn't been integrated yet.
> >>
> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >>
> >>> Hi all,
> >>> after reviewing some pull requests, I'm since a couple of days unable
> >>> to run the testsuite; since Anna's fixes affect many modules I'm
> >>> trying to run the testsuite of the whole project, as we should always
> >>> do but I admit I haven't done it in a while because of the core module
> >>> failures.
> >>>
> >>> So I run:
> >>> $ mvn -fn clean install
> >>>
> >>> using -fn to have it continue after the core failures.
> >>>
> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >>> sure this was good enough some months back.
> >>>
> >>> Second attempt slowed down like crazy, and I found a warning about
> >>> having filled the code cache size, so doubled it to 200M.
> >>>
> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> >>> -XX:MaxPermSize=380M which should be plenty?
> >>>
> >>> This is :
> >>> java version "1.6.0_43"
> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >>>
> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >>> -XX:ReservedCodeCacheSize=200M
> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >>>
> >>> My custom log configuration just disables trace & debug.
> >>>
> >>> Going to try now with larger PermGen and different JVMs but it looks
> >>> quite bad.. any other suggestion?
> >>> (I do have the security limits setup properly)
> >>>
> >>> Sanne
> >>> ___
> >>> infinispan-dev mailing list
> >>> infinispan-dev@lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> Cheers,
> >> --
> >> Mircea Markus
> >> Infinispan lead (www.infinispan.org)
> >>
> >>
> >>
> >>
> >>
> >> ___
> >> infinispan-dev mailing list
> >> infinispan-dev@lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> 
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)





___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Tristan Tarrant]

On 03/20/2013 04:12 PM, Dan Berindei wrote:
> The problem is that we still leak threads in almost every module, and 
> that means we keep a copy of the core classes (and all their 
> dependencies) for every module. Of course, some modules' dependencies 
> are already oversized, so keeping only one copy is already too much...
I did do quite a bit of cleanup in the server-core and server-hotrod 
modules a while back, but things might have regressed since then. I 
think we need to go on a cleaning spree.

Tristan
___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Dan Berindei]

On Wed, Mar 20, 2013 at 5:33 PM, Mircea Markus  wrote:

>
> On 20 Mar 2013, at 15:12, Dan Berindei wrote:
>
> > The problem is that we still leak threads in almost every module, and
> that means we keep a copy of the core classes (and all their dependencies)
> for every module. Of course, some modules' dependencies are already
> oversized, so keeping only one copy is already too much...
> do we have a JIRA for this?
>
>
There is https://issues.jboss.org/browse/ISPN-2477, but we probably need a
new issue to cover the test suite specifically.

>
> > I admit I don't run the whole test suite too often either, but I
> recently changed the Cloudbees settings to get rid of the OOM there. It
> uses about 550MB of permgen by the end of the test suite, without
> -XX:+UseCompressedOops. These are the settings I used:
> >
> > -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
> -XX:MaxPermSize=700M
>
> Thanks for sharing this Dan!
>
> >
> >
> > Cheers
> > Dan
> >
> >
> >
> > On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant 
> wrote:
> > Sanne, turn on CompressedOops ? Still those requirements are indeed
> > ridiculous.
> >
> > Tristan
> >
> > On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> > > I'm testing master, at da5c3f0
> > >
> > > Just killed a run which was using
> > >
> > > java version "1.7.0_17"
> > > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> > > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> > >
> > > this time again an OOM (while I have 2GB !), last sign of life came
> > > from the "Rolling Upgrade Tooling"
> > >
> > > I'm not going to merge/review any pull request until this works.
> > >
> > > Sanne
> > >
> > > On 20 March 2013 12:09, Mircea Markus  wrote:
> > >> I've just run it on master and didn't get OOM. well I'm using osx.
> Are you running it on master or a particular branch? Which module crashes?
> > >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
> the reason it hasn't been integrated yet.
> > >>
> > >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> > >>
> > >>> Hi all,
> > >>> after reviewing some pull requests, I'm since a couple of days unable
> > >>> to run the testsuite; since Anna's fixes affect many modules I'm
> > >>> trying to run the testsuite of the whole project, as we should always
> > >>> do but I admit I haven't done it in a while because of the core
> module
> > >>> failures.
> > >>>
> > >>> So I run:
> > >>> $ mvn -fn clean install
> > >>>
> > >>> using -fn to have it continue after the core failures.
> > >>>
> > >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> > >>> sure this was good enough some months back.
> > >>>
> > >>> Second attempt slowed down like crazy, and I found a warning about
> > >>> having filled the code cache size, so doubled it to 200M.
> > >>>
> > >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> > >>> -XX:MaxPermSize=380M which should be plenty?
> > >>>
> > >>> This is :
> > >>> java version "1.6.0_43"
> > >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> > >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> > >>>
> > >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> > >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> > >>> -XX:ReservedCodeCacheSize=200M
> > >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> > >>>
> > >>> My custom log configuration just disables trace & debug.
> > >>>
> > >>> Going to try now with larger PermGen and different JVMs but it looks
> > >>> quite bad.. any other suggestion?
> > >>> (I do have the security limits setup properly)
> > >>>
> > >>> Sanne
> > >>> ___
> > >>> infinispan-dev mailing list
> > >>> infinispan-dev@lists.jboss.org
> > >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >> Cheers,
> > >> --
> > >> Mircea Markus
> > >> Infinispan lead (www.infinispan.org)
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> ___
> > >> infinispan-dev mailing list
> > >> infinispan-dev@lists.jboss.org
> > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > > ___
> > > infinispan-dev mailing list
> > > infinispan-dev@lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > >
> >
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > ___
> > infinispan-dev mailing list
> > infinispan-dev@lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list

Re: [infinispan-dev] How to run the testsuite?

[Manik Surtani]

On 20 Mar 2013, at 15:29, Adrian Nistor  wrote:

> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  Previously 
> I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
> 
> Haven't tried to analyze closely the heap yet but first thing I noticed is 
> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
> probably is not the root cause of this issue, but is odd anyway :).

Yes, very odd.  Do you also see 19 instances of a GlobalComponentRegistry?

> 
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that 
>> means we keep a copy of the core classes (and all their dependencies) for 
>> every module. Of course, some modules' dependencies are already oversized, 
>> so keeping only one copy is already too much...
>> 
>> I admit I don't run the whole test suite too often either, but I recently 
>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>> 550MB of permgen by the end of the test suite, without 
>> -XX:+UseCompressedOops. These are the settings I used:
>> 
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>> -XX:MaxPermSize=700M
>> 
>> 
>> Cheers
>> Dan
>> 
>> 
>> 
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>> 
>> Tristan
>> 
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>> > I'm testing master, at da5c3f0
>> >
>> > Just killed a run which was using
>> >
>> > java version "1.7.0_17"
>> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>> >
>> > this time again an OOM (while I have 2GB !), last sign of life came
>> > from the "Rolling Upgrade Tooling"
>> >
>> > I'm not going to merge/review any pull request until this works.
>> >
>> > Sanne
>> >
>> > On 20 March 2013 12:09, Mircea Markus  wrote:
>> >> I've just run it on master and didn't get OOM. well I'm using osx. Are 
>> >> you running it on master or a particular branch? Which module crashes?
>> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
>> >> reason it hasn't been integrated yet.
>> >>
>> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> >>
>> >>> Hi all,
>> >>> after reviewing some pull requests, I'm since a couple of days unable
>> >>> to run the testsuite; since Anna's fixes affect many modules I'm
>> >>> trying to run the testsuite of the whole project, as we should always
>> >>> do but I admit I haven't done it in a while because of the core module
>> >>> failures.
>> >>>
>> >>> So I run:
>> >>> $ mvn -fn clean install
>> >>>
>> >>> using -fn to have it continue after the core failures.
>> >>>
>> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> >>> sure this was good enough some months back.
>> >>>
>> >>> Second attempt slowed down like crazy, and I found a warning about
>> >>> having filled the code cache size, so doubled it to 200M.
>> >>>
>> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> >>> -XX:MaxPermSize=380M which should be plenty?
>> >>>
>> >>> This is :
>> >>> java version "1.6.0_43"
>> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> >>>
>> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> >>> -XX:ReservedCodeCacheSize=200M
>> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> >>>
>> >>> My custom log configuration just disables trace & debug.
>> >>>
>> >>> Going to try now with larger PermGen and different JVMs but it looks
>> >>> quite bad.. any other suggestion?
>> >>> (I do have the security limits setup properly)
>> >>>
>> >>> Sanne
>> >>> ___
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev@lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> Cheers,
>> >> --
>> >> Mircea Markus
>> >> Infinispan lead (www.infinispan.org)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > ___
>> > infinispan-dev mailing list
>> > infinispan-dev@lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> 
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> 
>> __

Re: [infinispan-dev] How to run the testsuite?

[Sanne Grinovero]

Thanks Dan,
with the following options it completed the build:

MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
-Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
-Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
-XX:ReservedCodeCacheSize=200M
-Dlog4j.configuration=file:/opt/infinispan-log4j.xml

Sanne

On 20 March 2013 17:36, Manik Surtani  wrote:
>
> On 20 Mar 2013, at 15:29, Adrian Nistor  wrote:
>
> I've also tried changing the fork mode of surefire from 'none' to 'once' and
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
> Previously I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
>
> Haven't tried to analyze closely the heap yet but first thing I noticed is
> 15% of it is occupied by 19 ComponentMetadataRepo instances, which
> probably is not the root cause of this issue, but is odd anyway :).
>
>
> Yes, very odd.  Do you also see 19 instances of a
> GlobalComponentRegistry?
>
>
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>
> The problem is that we still leak threads in almost every module, and that
> means we keep a copy of the core classes (and all their dependencies) for
> every module. Of course, some modules' dependencies are already oversized,
> so keeping only one copy is already too much...
>
> I admit I don't run the whole test suite too often either, but I recently
> changed the Cloudbees settings to get rid of the OOM there. It uses about
> 550MB of permgen by the end of the test suite, without
> -XX:+UseCompressedOops. These are the settings I used:
>
> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
> -XX:MaxPermSize=700M
>
>
> Cheers
> Dan
>
>
>
> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant 
> wrote:
>>
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>>
>> Tristan
>>
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>> > I'm testing master, at da5c3f0
>> >
>> > Just killed a run which was using
>> >
>> > java version "1.7.0_17"
>> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>> >
>> > this time again an OOM (while I have 2GB !), last sign of life came
>> > from the "Rolling Upgrade Tooling"
>> >
>> > I'm not going to merge/review any pull request until this works.
>> >
>> > Sanne
>> >
>> > On 20 March 2013 12:09, Mircea Markus  wrote:
>> >> I've just run it on master and didn't get OOM. well I'm using osx. Are
>> >> you running it on master or a particular branch? Which module crashes?
>> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
>> >> the reason it hasn't been integrated yet.
>> >>
>> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> >>
>> >>> Hi all,
>> >>> after reviewing some pull requests, I'm since a couple of days unable
>> >>> to run the testsuite; since Anna's fixes affect many modules I'm
>> >>> trying to run the testsuite of the whole project, as we should always
>> >>> do but I admit I haven't done it in a while because of the core module
>> >>> failures.
>> >>>
>> >>> So I run:
>> >>> $ mvn -fn clean install
>> >>>
>> >>> using -fn to have it continue after the core failures.
>> >>>
>> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> >>> sure this was good enough some months back.
>> >>>
>> >>> Second attempt slowed down like crazy, and I found a warning about
>> >>> having filled the code cache size, so doubled it to 200M.
>> >>>
>> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> >>> -XX:MaxPermSize=380M which should be plenty?
>> >>>
>> >>> This is :
>> >>> java version "1.6.0_43"
>> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> >>>
>> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> >>> -XX:ReservedCodeCacheSize=200M
>> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> >>>
>> >>> My custom log configuration just disables trace & debug.
>> >>>
>> >>> Going to try now with larger PermGen and different JVMs but it looks
>> >>> quite bad.. any other suggestion?
>> >>> (I do have the security limits setup properly)
>> >>>
>> >>> Sanne
>> >>> ___
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev@lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> Cheers,
>> >> --
>> >> Mircea Markus
>> >> Infinispan lead (www.infinispan.org)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/li

Re: [infinispan-dev] How to run the testsuite?

[Dan Berindei]

Sanne, you Xmx setting seems a bit excessive :)

Did you see it fail on your machine with 1GB regular heap and 700MB permgen?


On Thu, Mar 21, 2013 at 1:55 AM, Sanne Grinovero wrote:

> Thanks Dan,
> with the following options it completed the build:
>
> MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
> -Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> -XX:ReservedCodeCacheSize=200M
> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>
> Sanne
>
> On 20 March 2013 17:36, Manik Surtani  wrote:
> >
> > On 20 Mar 2013, at 15:29, Adrian Nistor  wrote:
> >
> > I've also tried changing the fork mode of surefire from 'none' to 'once'
> and
> > the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
> > Previously I did not complete, 500mb was not enough.
> > Anyone knows why surefire was not allowed to fork?
> >
> > Haven't tried to analyze closely the heap yet but first thing I noticed
> is
> > 15% of it is occupied by 19 ComponentMetadataRepo instances, which
> > probably is not the root cause of this issue, but is odd anyway :).
> >
> >
> > Yes, very odd.  Do you also see 19 instances of a
> > GlobalComponentRegistry?
> >
> >
> > On 03/20/2013 05:12 PM, Dan Berindei wrote:
> >
> > The problem is that we still leak threads in almost every module, and
> that
> > means we keep a copy of the core classes (and all their dependencies) for
> > every module. Of course, some modules' dependencies are already
> oversized,
> > so keeping only one copy is already too much...
> >
> > I admit I don't run the whole test suite too often either, but I recently
> > changed the Cloudbees settings to get rid of the OOM there. It uses about
> > 550MB of permgen by the end of the test suite, without
> > -XX:+UseCompressedOops. These are the settings I used:
> >
> > -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
> > -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
> > -XX:MaxPermSize=700M
> >
> >
> > Cheers
> > Dan
> >
> >
> >
> > On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant 
> > wrote:
> >>
> >> Sanne, turn on CompressedOops ? Still those requirements are indeed
> >> ridiculous.
> >>
> >> Tristan
> >>
> >> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> >> > I'm testing master, at da5c3f0
> >> >
> >> > Just killed a run which was using
> >> >
> >> > java version "1.7.0_17"
> >> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> >> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> >> >
> >> > this time again an OOM (while I have 2GB !), last sign of life came
> >> > from the "Rolling Upgrade Tooling"
> >> >
> >> > I'm not going to merge/review any pull request until this works.
> >> >
> >> > Sanne
> >> >
> >> > On 20 March 2013 12:09, Mircea Markus  wrote:
> >> >> I've just run it on master and didn't get OOM. well I'm using osx.
> Are
> >> >> you running it on master or a particular branch? Which module
> crashes?
> >> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
> >> >> the reason it hasn't been integrated yet.
> >> >>
> >> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> >> >>
> >> >>> Hi all,
> >> >>> after reviewing some pull requests, I'm since a couple of days
> unable
> >> >>> to run the testsuite; since Anna's fixes affect many modules I'm
> >> >>> trying to run the testsuite of the whole project, as we should
> always
> >> >>> do but I admit I haven't done it in a while because of the core
> module
> >> >>> failures.
> >> >>>
> >> >>> So I run:
> >> >>> $ mvn -fn clean install
> >> >>>
> >> >>> using -fn to have it continue after the core failures.
> >> >>>
> >> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
> >> >>> sure this was good enough some months back.
> >> >>>
> >> >>> Second attempt slowed down like crazy, and I found a warning about
> >> >>> having filled the code cache size, so doubled it to 200M.
> >> >>>
> >> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
> >> >>> -XX:MaxPermSize=380M which should be plenty?
> >> >>>
> >> >>> This is :
> >> >>> java version "1.6.0_43"
> >> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
> >> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
> >> >>>
> >> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
> >> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> >> >>> -XX:ReservedCodeCacheSize=200M
> >> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> >> >>>
> >> >>> My custom log configuration just disables trace & debug.
> >> >>>
> >> >>> Going to try now with larger PermGen and different JVMs but it looks
> >> >>> quite bad.. any other suggestion?
> >> >>> (I do have the security limits setup properly)
> >> >>>
> >> >>> Sanne
> >> >>> ___

Re: [infinispan-dev] How to run the testsuite?

[Galder Zamarreño]

On Mar 20, 2013, at 3:29 PM, Adrian Nistor  wrote:

> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  Previously 
> I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
> 
> Haven't tried to analyze closely the heap yet but first thing I noticed is 
> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
> probably is not the root cause of this issue, but is odd anyway :).

If you have a heap dump, can you put it somewhere online so that we can inspect 
it? Eclipse MAT is great for that...

> 
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that 
>> means we keep a copy of the core classes (and all their dependencies) for 
>> every module. Of course, some modules' dependencies are already oversized, 
>> so keeping only one copy is already too much...
>> 
>> I admit I don't run the whole test suite too often either, but I recently 
>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>> 550MB of permgen by the end of the test suite, without 
>> -XX:+UseCompressedOops. These are the settings I used:
>> 
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>> -XX:MaxPermSize=700M
>> 
>> 
>> Cheers
>> Dan
>> 
>> 
>> 
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>> 
>> Tristan
>> 
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>> > I'm testing master, at da5c3f0
>> >
>> > Just killed a run which was using
>> >
>> > java version "1.7.0_17"
>> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>> >
>> > this time again an OOM (while I have 2GB !), last sign of life came
>> > from the "Rolling Upgrade Tooling"
>> >
>> > I'm not going to merge/review any pull request until this works.
>> >
>> > Sanne
>> >
>> > On 20 March 2013 12:09, Mircea Markus  wrote:
>> >> I've just run it on master and didn't get OOM. well I'm using osx. Are 
>> >> you running it on master or a particular branch? Which module crashes?
>> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
>> >> reason it hasn't been integrated yet.
>> >>
>> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> >>
>> >>> Hi all,
>> >>> after reviewing some pull requests, I'm since a couple of days unable
>> >>> to run the testsuite; since Anna's fixes affect many modules I'm
>> >>> trying to run the testsuite of the whole project, as we should always
>> >>> do but I admit I haven't done it in a while because of the core module
>> >>> failures.
>> >>>
>> >>> So I run:
>> >>> $ mvn -fn clean install
>> >>>
>> >>> using -fn to have it continue after the core failures.
>> >>>
>> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> >>> sure this was good enough some months back.
>> >>>
>> >>> Second attempt slowed down like crazy, and I found a warning about
>> >>> having filled the code cache size, so doubled it to 200M.
>> >>>
>> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> >>> -XX:MaxPermSize=380M which should be plenty?
>> >>>
>> >>> This is :
>> >>> java version "1.6.0_43"
>> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> >>>
>> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> >>> -XX:ReservedCodeCacheSize=200M
>> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> >>>
>> >>> My custom log configuration just disables trace & debug.
>> >>>
>> >>> Going to try now with larger PermGen and different JVMs but it looks
>> >>> quite bad.. any other suggestion?
>> >>> (I do have the security limits setup properly)
>> >>>
>> >>> Sanne
>> >>> ___
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev@lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> Cheers,
>> >> --
>> >> Mircea Markus
>> >> Infinispan lead (www.infinispan.org)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > ___
>> > infinispan-dev mailing list
>> > infinispan-dev@lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> 
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> 

Re: [infinispan-dev] How to run the testsuite?

[Adrian Nistor]

A correction is needed. I realized later that those 19000 instances were 
actually ComponentMetadata not ComponentMetadataRepo!
There were a few hundred ComponentMetadataRepo instances and the same 
number of GlobalComponentRegistry - nothing very unusual. I counted them 
using MAT but I lost the hprof files.

On 03/21/2013 01:35 PM, Galder Zamarreño wrote:
> On Mar 20, 2013, at 3:29 PM, Adrian Nistor  wrote:
>
>> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  
>> Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>>
>> Haven't tried to analyze closely the heap yet but first thing I noticed is 
>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
>> probably is not the root cause of this issue, but is odd anyway :).
> If you have a heap dump, can you put it somewhere online so that we can 
> inspect it? Eclipse MAT is great for that...
>
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>> The problem is that we still leak threads in almost every module, and that 
>>> means we keep a copy of the core classes (and all their dependencies) for 
>>> every module. Of course, some modules' dependencies are already oversized, 
>>> so keeping only one copy is already too much...
>>>
>>> I admit I don't run the whole test suite too often either, but I recently 
>>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>>> 550MB of permgen by the end of the test suite, without 
>>> -XX:+UseCompressedOops. These are the settings I used:
>>>
>>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>>> -XX:MaxPermSize=700M
>>>
>>>
>>> Cheers
>>> Dan
>>>
>>>
>>>
>>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  
>>> wrote:
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>>
>>> Tristan
>>>
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
 I'm testing master, at da5c3f0

 Just killed a run which was using

 java version "1.7.0_17"
 Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

 this time again an OOM (while I have 2GB !), last sign of life came
 from the "Rolling Upgrade Tooling"

 I'm not going to merge/review any pull request until this works.

 Sanne

 On 20 March 2013 12:09, Mircea Markus  wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are 
> you running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
> reason it hasn't been integrated yet.
>
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>>
>> So I run:
>> $ mvn -fn clean install
>>
>> using -fn to have it continue after the core failures.
>>
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>>
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>>
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>>
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>
>> My custom log configuration just disables trace & debug.
>>
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>>
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> ___
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo

Re: [infinispan-dev] How to run the testsuite?

[Adrian Nistor]

If anyone wants to analyze the heap dump, I've recreated it: 
http://dl.dropbox.com/u/29415206/java_pid27113.zip

On 03/25/2013 04:47 PM, Adrian Nistor wrote:
> A correction is needed. I realized later that those 19000 instances were
> actually ComponentMetadata not ComponentMetadataRepo!
> There were a few hundred ComponentMetadataRepo instances and the same
> number of GlobalComponentRegistry - nothing very unusual. I counted them
> using MAT but I lost the hprof files.
>
> On 03/21/2013 01:35 PM, Galder Zamarreño wrote:
>> On Mar 20, 2013, at 3:29 PM, Adrian Nistor  wrote:
>>
>>> I've also tried changing the fork mode of surefire from 'none' to 'once' 
>>> and the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  
>>> Previously I did not complete, 500mb was not enough.
>>> Anyone knows why surefire was not allowed to fork?
>>>
>>> Haven't tried to analyze closely the heap yet but first thing I noticed is 
>>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
>>> probably is not the root cause of this issue, but is odd anyway :).
>> If you have a heap dump, can you put it somewhere online so that we can 
>> inspect it? Eclipse MAT is great for that...
>>
>>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
 The problem is that we still leak threads in almost every module, and that 
 means we keep a copy of the core classes (and all their dependencies) for 
 every module. Of course, some modules' dependencies are already oversized, 
 so keeping only one copy is already too much...

 I admit I don't run the whole test suite too often either, but I recently 
 changed the Cloudbees settings to get rid of the OOM there. It uses about 
 550MB of permgen by the end of the test suite, without 
 -XX:+UseCompressedOops. These are the settings I used:

 -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
 -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
 -XX:MaxPermSize=700M


 Cheers
 Dan



 On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  
 wrote:
 Sanne, turn on CompressedOops ? Still those requirements are indeed
 ridiculous.

 Tristan

 On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
>
> Just killed a run which was using
>
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
>
> I'm not going to merge/review any pull request until this works.
>
> Sanne
>
> On 20 March 2013 12:09, Mircea Markus  wrote:
>> I've just run it on master and didn't get OOM. well I'm using osx. Are 
>> you running it on master or a particular branch? Which module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
>> reason it hasn't been integrated yet.
>>
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should always
>>> do but I admit I haven't done it in a while because of the core module
>>> failures.
>>>
>>> So I run:
>>> $ mvn -fn clean install
>>>
>>> using -fn to have it continue after the core failures.
>>>
>>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>>> sure this was good enough some months back.
>>>
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>>
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>>
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>>
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>>
>>> My custom log configuration just disables trace & debug.
>>>
>>> Going to try now with larger PermGen and different JVMs but it looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>>
>>> Sanne
>>> ___
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> Chee

Re: [infinispan-dev] How to run the testsuite?

[Mircea Markus]

I've upgrade to mvn 2.14 and configured: forkCount=1/reuseForks=true (default 
settings)  "which means that Surefire creates *one new JVM* process to execute 
all tests in one *maven module*." [1]

It doesn't do that: it reuses the same process to execute all the tests in 
*all* modules. 

[1] 
http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

On 20 Mar 2013, at 15:29, Adrian Nistor wrote:

> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  Previously 
> I did not complete, 500mb was not enough.
> Anyone knows why surefire was not allowed to fork?
> 
> Haven't tried to analyze closely the heap yet but first thing I noticed is 
> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
> probably is not the root cause of this issue, but is odd anyway :).
> 
> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> The problem is that we still leak threads in almost every module, and that 
>> means we keep a copy of the core classes (and all their dependencies) for 
>> every module. Of course, some modules' dependencies are already oversized, 
>> so keeping only one copy is already too much...
>> 
>> I admit I don't run the whole test suite too often either, but I recently 
>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>> 550MB of permgen by the end of the test suite, without 
>> -XX:+UseCompressedOops. These are the settings I used:
>> 
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>> -XX:MaxPermSize=700M
>> 
>> 
>> Cheers
>> Dan
>> 
>> 
>> 
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  wrote:
>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>> ridiculous.
>> 
>> Tristan
>> 
>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
>> > I'm testing master, at da5c3f0
>> >
>> > Just killed a run which was using
>> >
>> > java version "1.7.0_17"
>> > Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
>> >
>> > this time again an OOM (while I have 2GB !), last sign of life came
>> > from the "Rolling Upgrade Tooling"
>> >
>> > I'm not going to merge/review any pull request until this works.
>> >
>> > Sanne
>> >
>> > On 20 March 2013 12:09, Mircea Markus  wrote:
>> >> I've just run it on master and didn't get OOM. well I'm using osx. Are 
>> >> you running it on master or a particular branch? Which module crashes?
>> >> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
>> >> reason it hasn't been integrated yet.
>> >>
>> >> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> >>
>> >>> Hi all,
>> >>> after reviewing some pull requests, I'm since a couple of days unable
>> >>> to run the testsuite; since Anna's fixes affect many modules I'm
>> >>> trying to run the testsuite of the whole project, as we should always
>> >>> do but I admit I haven't done it in a while because of the core module
>> >>> failures.
>> >>>
>> >>> So I run:
>> >>> $ mvn -fn clean install
>> >>>
>> >>> using -fn to have it continue after the core failures.
>> >>>
>> >>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> >>> sure this was good enough some months back.
>> >>>
>> >>> Second attempt slowed down like crazy, and I found a warning about
>> >>> having filled the code cache size, so doubled it to 200M.
>> >>>
>> >>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> >>> -XX:MaxPermSize=380M which should be plenty?
>> >>>
>> >>> This is :
>> >>> java version "1.6.0_43"
>> >>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> >>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> >>>
>> >>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> >>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> >>> -XX:ReservedCodeCacheSize=200M
>> >>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> >>>
>> >>> My custom log configuration just disables trace & debug.
>> >>>
>> >>> Going to try now with larger PermGen and different JVMs but it looks
>> >>> quite bad.. any other suggestion?
>> >>> (I do have the security limits setup properly)
>> >>>
>> >>> Sanne
>> >>> ___
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev@lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> Cheers,
>> >> --
>> >> Mircea Markus
>> >> Infinispan lead (www.infinispan.org)
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ___
>> >> infinispan-dev mailing list
>> >> infinispan-dev@lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > ___
>> > infinispan-dev mailing list
>> > infinispan

Re: [infinispan-dev] How to run the testsuite?

[Manik Surtani]

On 5 Apr 2013, at 17:01, Mircea Markus  wrote:

> I've upgrade to mvn 2.14

You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure we 
enforce mvn 3.x.  ;)

> and configured: forkCount=1/reuseForks=true (default settings)  "which means 
> that Surefire creates *one new JVM* process to execute all tests in one 
> *maven module*." [1]
> 
> It doesn't do that: it reuses the same process to execute all the tests in 
> *all* modules. 
> 
> [1] 
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
> 
> On 20 Mar 2013, at 15:29, Adrian Nistor wrote:
> 
>> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  
>> Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>> 
>> Haven't tried to analyze closely the heap yet but first thing I noticed is 
>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
>> probably is not the root cause of this issue, but is odd anyway :).
>> 
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>> The problem is that we still leak threads in almost every module, and that 
>>> means we keep a copy of the core classes (and all their dependencies) for 
>>> every module. Of course, some modules' dependencies are already oversized, 
>>> so keeping only one copy is already too much...
>>> 
>>> I admit I don't run the whole test suite too often either, but I recently 
>>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>>> 550MB of permgen by the end of the test suite, without 
>>> -XX:+UseCompressedOops. These are the settings I used:
>>> 
>>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>>> -XX:MaxPermSize=700M
>>> 
>>> 
>>> Cheers
>>> Dan
>>> 
>>> 
>>> 
>>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  
>>> wrote:
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>> 
>>> Tristan
>>> 
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
 I'm testing master, at da5c3f0
 
 Just killed a run which was using
 
 java version "1.7.0_17"
 Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
 
 this time again an OOM (while I have 2GB !), last sign of life came
 from the "Rolling Upgrade Tooling"
 
 I'm not going to merge/review any pull request until this works.
 
 Sanne
 
 On 20 March 2013 12:09, Mircea Markus  wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are 
> you running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
> reason it hasn't been integrated yet.
> 
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> 
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>> 
>> So I run:
>> $ mvn -fn clean install
>> 
>> using -fn to have it continue after the core failures.
>> 
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>> 
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>> 
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>> 
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> 
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> 
>> My custom log configuration just disables trace & debug.
>> 
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>> 
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
> 
> 
> 
> 
> 
> __

Re: [infinispan-dev] How to run the testsuite?

[Manik Surtani]

On 5 Apr 2013, at 17:01, Mircea Markus  wrote:

> I've upgrade to mvn 2.14

You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure we 
enforce mvn 3.x.  ;)

> and configured: forkCount=1/reuseForks=true (default settings)  "which means 
> that Surefire creates *one new JVM* process to execute all tests in one 
> *maven module*." [1]
> 
> It doesn't do that: it reuses the same process to execute all the tests in 
> *all* modules. 
> 
> [1] 
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
> 
> On 20 Mar 2013, at 15:29, Adrian Nistor wrote:
> 
>> I've also tried changing the fork mode of surefire from 'none' to 'once' and 
>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.  
>> Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>> 
>> Haven't tried to analyze closely the heap yet but first thing I noticed is 
>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which 
>> probably is not the root cause of this issue, but is odd anyway :).
>> 
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>> The problem is that we still leak threads in almost every module, and that 
>>> means we keep a copy of the core classes (and all their dependencies) for 
>>> every module. Of course, some modules' dependencies are already oversized, 
>>> so keeping only one copy is already too much...
>>> 
>>> I admit I don't run the whole test suite too often either, but I recently 
>>> changed the Cloudbees settings to get rid of the OOM there. It uses about 
>>> 550MB of permgen by the end of the test suite, without 
>>> -XX:+UseCompressedOops. These are the settings I used:
>>> 
>>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC 
>>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m 
>>> -XX:MaxPermSize=700M
>>> 
>>> 
>>> Cheers
>>> Dan
>>> 
>>> 
>>> 
>>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant  
>>> wrote:
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>> 
>>> Tristan
>>> 
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
 I'm testing master, at da5c3f0
 
 Just killed a run which was using
 
 java version "1.7.0_17"
 Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
 
 this time again an OOM (while I have 2GB !), last sign of life came
 from the "Rolling Upgrade Tooling"
 
 I'm not going to merge/review any pull request until this works.
 
 Sanne
 
 On 20 March 2013 12:09, Mircea Markus  wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are 
> you running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's the 
> reason it hasn't been integrated yet.
> 
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> 
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>> 
>> So I run:
>> $ mvn -fn clean install
>> 
>> using -fn to have it continue after the core failures.
>> 
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>> 
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>> 
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>> 
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> 
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> 
>> My custom log configuration just disables trace & debug.
>> 
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>> 
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
> 
> 
> 
> 
> 
> __

Re: [infinispan-dev] How to run the testsuite?

[Mircea Markus]

On 8 Apr 2013, at 10:02, Manik Surtani wrote:

>> I've upgrade to mvn 2.14
> 
> You mean Surefire 2.14. You had me confused for a bit, since I'm pretty sure 
> we enforce mvn 3.x.  ;)
yep, surefire 2.14 :-)

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)





___
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev

Re: [infinispan-dev] How to run the testsuite?

[Mircea Markus]

Now the suite is forks a new process for each module. I've complete the run 
with 128m MaxPerm.

On 20 Mar 2013, at 23:55, Sanne Grinovero wrote:
> Thanks Dan,
> with the following options it completed the build:
> 
> MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
> -Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
> -XX:ReservedCodeCacheSize=200M
> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
> 
> Sanne
> 
> On 20 March 2013 17:36, Manik Surtani  wrote:
>> 
>> On 20 Mar 2013, at 15:29, Adrian Nistor  wrote:
>> 
>> I've also tried changing the fork mode of surefire from 'none' to 'once' and
>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
>> Previously I did not complete, 500mb was not enough.
>> Anyone knows why surefire was not allowed to fork?
>> 
>> Haven't tried to analyze closely the heap yet but first thing I noticed is
>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which
>> probably is not the root cause of this issue, but is odd anyway :).
>> 
>> 
>> Yes, very odd.  Do you also see 19 instances of a
>> GlobalComponentRegistry?
>> 
>> 
>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>> 
>> The problem is that we still leak threads in almost every module, and that
>> means we keep a copy of the core classes (and all their dependencies) for
>> every module. Of course, some modules' dependencies are already oversized,
>> so keeping only one copy is already too much...
>> 
>> I admit I don't run the whole test suite too often either, but I recently
>> changed the Cloudbees settings to get rid of the OOM there. It uses about
>> 550MB of permgen by the end of the test suite, without
>> -XX:+UseCompressedOops. These are the settings I used:
>> 
>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
>> -XX:MaxPermSize=700M
>> 
>> 
>> Cheers
>> Dan
>> 
>> 
>> 
>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant 
>> wrote:
>>> 
>>> Sanne, turn on CompressedOops ? Still those requirements are indeed
>>> ridiculous.
>>> 
>>> Tristan
>>> 
>>> On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
 I'm testing master, at da5c3f0
 
 Just killed a run which was using
 
 java version "1.7.0_17"
 Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
 
 this time again an OOM (while I have 2GB !), last sign of life came
 from the "Rolling Upgrade Tooling"
 
 I'm not going to merge/review any pull request until this works.
 
 Sanne
 
 On 20 March 2013 12:09, Mircea Markus  wrote:
> I've just run it on master and didn't get OOM. well I'm using osx. Are
> you running it on master or a particular branch? Which module crashes?
> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
> the reason it hasn't been integrated yet.
> 
> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
> 
>> Hi all,
>> after reviewing some pull requests, I'm since a couple of days unable
>> to run the testsuite; since Anna's fixes affect many modules I'm
>> trying to run the testsuite of the whole project, as we should always
>> do but I admit I haven't done it in a while because of the core module
>> failures.
>> 
>> So I run:
>> $ mvn -fn clean install
>> 
>> using -fn to have it continue after the core failures.
>> 
>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>> sure this was good enough some months back.
>> 
>> Second attempt slowed down like crazy, and I found a warning about
>> having filled the code cache size, so doubled it to 200M.
>> 
>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>> -XX:MaxPermSize=380M which should be plenty?
>> 
>> This is :
>> java version "1.6.0_43"
>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>> 
>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> 
>> My custom log configuration just disables trace & debug.
>> 
>> Going to try now with larger PermGen and different JVMs but it looks
>> quite bad.. any other suggestion?
>> (I do have the security limits setup properly)
>> 
>> Sanne
>> ___
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> Cheers,
> --
>>

Re: [infinispan-dev] How to run the testsuite?

[Galder Zamarreño]

Nice :)

On Apr 18, 2013, at 6:43 AM, Mircea Markus  wrote:

> Now the suite is forks a new process for each module. I've complete the run 
> with 128m MaxPerm.
> 
> On 20 Mar 2013, at 23:55, Sanne Grinovero wrote:
>> Thanks Dan,
>> with the following options it completed the build:
>> 
>> MAVEN_OPTS=-server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
>> -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:NewRatio=4 -Xss500k
>> -Xmx16G -Xms1G -XX:MaxPermSize=700M -XX:HeapDumpPath=/tmp/java_heap
>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>> -XX:ReservedCodeCacheSize=200M
>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>> 
>> Sanne
>> 
>> On 20 March 2013 17:36, Manik Surtani  wrote:
>>> 
>>> On 20 Mar 2013, at 15:29, Adrian Nistor  wrote:
>>> 
>>> I've also tried changing the fork mode of surefire from 'none' to 'once' and
>>> the entire suite runs fine now on jvm 1.6 with 500mb MaxPermSize.
>>> Previously I did not complete, 500mb was not enough.
>>> Anyone knows why surefire was not allowed to fork?
>>> 
>>> Haven't tried to analyze closely the heap yet but first thing I noticed is
>>> 15% of it is occupied by 19 ComponentMetadataRepo instances, which
>>> probably is not the root cause of this issue, but is odd anyway :).
>>> 
>>> 
>>> Yes, very odd.  Do you also see 19 instances of a
>>> GlobalComponentRegistry?
>>> 
>>> 
>>> On 03/20/2013 05:12 PM, Dan Berindei wrote:
>>> 
>>> The problem is that we still leak threads in almost every module, and that
>>> means we keep a copy of the core classes (and all their dependencies) for
>>> every module. Of course, some modules' dependencies are already oversized,
>>> so keeping only one copy is already too much...
>>> 
>>> I admit I don't run the whole test suite too often either, but I recently
>>> changed the Cloudbees settings to get rid of the OOM there. It uses about
>>> 550MB of permgen by the end of the test suite, without
>>> -XX:+UseCompressedOops. These are the settings I used:
>>> 
>>> -server -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
>>> -XX:+CMSClassUnloadingEnabled   -XX:NewRatio=4 -Xss500k -Xms100m -Xmx900m
>>> -XX:MaxPermSize=700M
>>> 
>>> 
>>> Cheers
>>> Dan
>>> 
>>> 
>>> 
>>> On Wed, Mar 20, 2013 at 2:59 PM, Tristan Tarrant 
>>> wrote:
 
 Sanne, turn on CompressedOops ? Still those requirements are indeed
 ridiculous.
 
 Tristan
 
 On 03/20/2013 01:27 PM, Sanne Grinovero wrote:
> I'm testing master, at da5c3f0
> 
> Just killed a run which was using
> 
> java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> 
> this time again an OOM (while I have 2GB !), last sign of life came
> from the "Rolling Upgrade Tooling"
> 
> I'm not going to merge/review any pull request until this works.
> 
> Sanne
> 
> On 20 March 2013 12:09, Mircea Markus  wrote:
>> I've just run it on master and didn't get OOM. well I'm using osx. Are
>> you running it on master or a particular branch? Which module crashes?
>> e.g. pedro's ISPN-2808 adds quite some threads to the party - that's
>> the reason it hasn't been integrated yet.
>> 
>> On 20 Mar 2013, at 11:40, Sanne Grinovero wrote:
>> 
>>> Hi all,
>>> after reviewing some pull requests, I'm since a couple of days unable
>>> to run the testsuite; since Anna's fixes affect many modules I'm
>>> trying to run the testsuite of the whole project, as we should always
>>> do but I admit I haven't done it in a while because of the core module
>>> failures.
>>> 
>>> So I run:
>>> $ mvn -fn clean install
>>> 
>>> using -fn to have it continue after the core failures.
>>> 
>>> First attempt gave me an OOM, was running with 1G heap.. I'm pretty
>>> sure this was good enough some months back.
>>> 
>>> Second attempt slowed down like crazy, and I found a warning about
>>> having filled the code cache size, so doubled it to 200M.
>>> 
>>> Third attempt: OutOfMemoryError: PermGen space! But I'm running with
>>> -XX:MaxPermSize=380M which should be plenty?
>>> 
>>> This is :
>>> java version "1.6.0_43"
>>> Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
>>> 
>>> MAVEN_OPTS=-Xmx2G -XX:MaxPermSize=380M -XX:+TieredCompilation
>>> -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
>>> -XX:ReservedCodeCacheSize=200M
>>> -Dlog4j.configuration=file:/opt/infinispan-log4j.xml
>>> 
>>> My custom log configuration just disables trace & debug.
>>> 
>>> Going to try now with larger PermGen and different JVMs but it looks
>>> quite bad.. any other suggestion?
>>> (I do have the security limits setup properly)
>>> 
>>> Sanne
>>> _