Re: Axis 2 Memory Leak when Client is instantiated within Service - pertains to edu.emory...ConcurrentHashMap -> WAS: Re: When do cleanup functions need to be called?
Matt, did you get this issue fixed? I am facing the same error in the project I am working on. I am using Axis 1.4.1. Appreciate your help. Regards Jiji Matt Friedman wrote: > > Hi, > > I'm sorry for the re-post, but I was discussing this thread with a > co-worker and we postulated that the title of the existing post (Re: > When do cleanup functions need to be called?), didn't represent what > this thread is actually about. > > I was afraid the importance of the thread might be missed, as the old > title no longer applies. > > Hopefully, this isn't inappropriate. Sorry if I've annoyed anyone. > > So, renamed more appropriately, here is the thread... > > Many thanks, > Matt Friedman. > > > 2008/9/23 Matt Friedman : >> Hi David, >> >> Thanks very much for this. >> >> When you say: "Try 1.4.1. It's much better than 1.4 with this >> particular memory leak.", I take it you mean it is partially fixed, >> but not entirely fixed in 1.4.1. Is that correct? >> >> Did your pool of stubs continue to work for you? >> >> My goal is to completely eliminate any mem leaks from our app. >> >> Thanks, >> Matt. >> >> >> 2008/9/23 David Rees : >>> On Mon, Sep 22, 2008 at 6:39 PM, Matt Friedman >>> wrote: >>>> Hi, was this issue fixed or resolved? I am running our service in >>>> JProbe and we are having the identical memory leak issues as described >>>> in this email chain. After roughly 900 requests the garbage collector >>>> goes to 100% and out of memory occurs. >>>> >>>> Over half the heap is full with instances related to: >>>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.* >>>> which is tied to the Axis2 implementation. We don't use edu.emory.* at >>>> all. >>>> >>>> We are running Axis2 v1.4. We're going to try a more recent release or >>>> the svn nightly build. >>> >>> Try 1.4.1. It's much better than 1.4 with this particular memory leak. >>> If your access to the stub constructor isn't synchronized, you may >>> need to serialize access to stub constructors as well. >>> >>> -Dave >>> >>> - >>> To unsubscribe, e-mail: axis-user-unsubscr...@ws.apache.org >>> For additional commands, e-mail: axis-user-h...@ws.apache.org >>> >>> >> >> >> >> -- >> -- Matt Friedman >> > > > > -- > -- Matt Friedman > > - > To unsubscribe, e-mail: axis-user-unsubscr...@ws.apache.org > For additional commands, e-mail: axis-user-h...@ws.apache.org > > > -- View this message in context: http://www.nabble.com/Axis-2-Memory-Leak-when-Client-is-instantiated-within-Service---pertains-to-edu.emory...ConcurrentHashMap--%3E-WAS%3A-Re%3A-When-do-cleanup-functions-need-to-be-called--tp19631299p23850641.html Sent from the Axis - User mailing list archive at Nabble.com.
Re: When do cleanup functions need to be called?
Amila Suriarachchi wrote: > > Please try with a Nighly build. As I think this could be the problem. > ... > -- > Amila Suriarachchi > WSO2 Inc. > blog: http://amilachinthaka.blogspot.com/ > Any idea when an release with this fix is planned ? Would this be in 1.4.2 ? -- View this message in context: http://www.nabble.com/When-do-cleanup-functions-need-to-be-called--tp18724207p20749331.html Sent from the Axis - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
Please try with a Nighly build. As I think this could be the problem.
in Stub class we have this at the clean up method.
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
_service.getAxisConfiguration().removeService(_service.getName());
}
this removes the service from the Axisconfiguration but does not remove the
end points.
If we take the ServiceClient clean up method. it calls to
axisConfiguration.removeServiceGroup(serviceGroupName);
this method looks like this,
Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) {
AxisService axisService = (AxisService) services.next();
allServices.remove(axisService.getName());
if (!axisService.isClientSide()) {
notifyObservers(AxisEvent.SERVICE_REMOVE, axisService);
}
//removes the endpoints to this service
String serviceName = axisService.getName();
String key = null;
for (Iterator iter =
axisService.getEndpoints().keySet().iterator(); iter.hasNext();){
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}
}
As you can see here Now the service is not available and as a result of that
endpoints are not removed.
I removed the service removal code from the Stub and as a result of that I
did see the memory leak with was there earlier.
Axis2 trunk has this change already.
thanks,
Amila.
On Tue, Oct 7, 2008 at 7:35 AM, Mel T <[EMAIL PROTECTED]> wrote:
>
>
>
> >> I have the same problem as described here.
> >> Upgrading to 1.4.1 has reduced the leak.
> >> I am now going to try the pool of stubs as well for the remaining.
> >>
> >> Dave, is your client running in Tomcat ?
> >> My problem only occurs when I am accessing the WS using a stub from
> >> within
> >> the same tomcat application.
> >
> > Yes, my client is running in Tomcat, Tomcat 5.5.27 to be exact. I
> > wonder if that is somehow also contributing to the issue?
> >
> > -Dave
> >
>
> I believe this is an issue with that setup. Not easy to debug. I will post
> here if I find something.
>
> --
> View this message in context:
> http://www.nabble.com/When-do-cleanup-functions-need-to-be-called--tp18724207p19849777.html
> Sent from the Axis - User mailing list archive at Nabble.com.
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: When do cleanup functions need to be called?
>> I have the same problem as described here. >> Upgrading to 1.4.1 has reduced the leak. >> I am now going to try the pool of stubs as well for the remaining. >> >> Dave, is your client running in Tomcat ? >> My problem only occurs when I am accessing the WS using a stub from >> within >> the same tomcat application. > > Yes, my client is running in Tomcat, Tomcat 5.5.27 to be exact. I > wonder if that is somehow also contributing to the issue? > > -Dave > I believe this is an issue with that setup. Not easy to debug. I will post here if I find something. -- View this message in context: http://www.nabble.com/When-do-cleanup-functions-need-to-be-called--tp18724207p19849777.html Sent from the Axis - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Mon, Oct 6, 2008 at 5:27 PM, Mel T <[EMAIL PROTECTED]> wrote: > I have the same problem as described here. > Upgrading to 1.4.1 has reduced the leak. > I am now going to try the pool of stubs as well for the remaining. > > Dave, is your client running in Tomcat ? > My problem only occurs when I am accessing the WS using a stub from within > the same tomcat application. Yes, my client is running in Tomcat, Tomcat 5.5.27 to be exact. I wonder if that is somehow also contributing to the issue? -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
I have the same problem as described here. Upgrading to 1.4.1 has reduced the leak. I am now going to try the pool of stubs as well for the remaining. Dave, is your client running in Tomcat ? My problem only occurs when I am accessing the WS using a stub from within the same tomcat application. -- Mel T. -- View this message in context: http://www.nabble.com/When-do-cleanup-functions-need-to-be-called--tp18724207p19848906.html Sent from the Axis - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Tue, Sep 23, 2008 at 5:16 AM, Matt Friedman <[EMAIL PROTECTED]> wrote: > When you say: "Try 1.4.1. It's much better than 1.4 with this > particular memory leak.", I take it you mean it is partially fixed, > but not entirely fixed in 1.4.1. Is that correct? > > Did your pool of stubs continue to work for you? > > My goal is to completely eliminate any mem leaks from our app. Yes, it has continued to work for me, and I don't know of any memory leaks. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Axis 2 Memory Leak when Client is instantiated within Service - pertains to edu.emory...ConcurrentHashMap -> WAS: Re: When do cleanup functions need to be called?
I'm really sorry, I didn't attach the whole thread the first time.
Here are the details...
--
Hi, was this issue fixed or resolved? I am running our service in
JProbe and we are having the identical memory leak issues as described
in this email chain. After roughly 900 requests the garbage collector
goes to 100% and out of memory occurs.
Over half the heap is full with instances related to:
edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.*
which is tied to the Axis2 implementation. We don't use edu.emory.* at
all.
We are running Axis2 v1.4. We're going to try a more recent release or
the svn nightly build.
Any thoughts or recommendations?
If you need any further information please let me know.
Many thanks,
Matt Friedman.
2008/7/31 Amila Suriarachchi <[EMAIL PROTECTED]>:
- Hide quoted text -
>
>
> On Wed, Jul 30, 2008 at 11:08 PM, David Rees <[EMAIL PROTECTED]> wrote:
>>
>> On Wed, Jul 30, 2008 at 4:22 AM, Amila Suriarachchi
>> <[EMAIL PROTECTED]> wrote:
>> > On Wed, Jul 30, 2008 at 2:15 PM, David Rees <[EMAIL PROTECTED]> wrote:
>> >
>> > well see this code,
>> >
>> > public static void clientCall1() {
>> > MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
>> > ClientCall1 req = new ClientCall1();
>> > ClientCall1Response res = stub.clientCall1(req);
>> > // Need to call this when calling using this function in a web service
>> > // Otherwise huge resource leak occurs
>> > stub._getServiceClient().
>> > cleanup();
>> > }
>> > }
>> >
>> > After this method call this class should be garbage collected. since
>> > stub
>> > does not have any reference after that.
>>
>> Right - but I'm not having a problem with Stubs being leaked. Here's a
>> count of various interesting classes (I've skipped some of the Java
>> core classes for brevity) of which there are a high number of
>> instances when the heap runs out of space (using a 32mb heap):
>>
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
>> 177,023, 17% heap
>>
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry[]
>> 177,023, 6% heap
>>
>> edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
>> 177,023, 8% heap
>> java.util.HashMap$Entry[] 37926, 8% heap
>> java.util.HashMap 36789, 4% heap
>> java.util.HashMap$Entry 28853, 2% heap
>> java.util.ArrayList 13698, 1% heap
>> org.apache.axis2.description.ParameterIncludeImpl 11099, 0% heap
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap 11064, 1%
>> heap
>> org.apache.axis2.description.PolicySubject 11063, 0% heap
>> org.apache.axis2.engine.Phase 7905, 0% heap
>> org.apache.axis2.description.AxisBindingMessage 4668, 1% heap
>> org.apache.axis2.description.AxisMessage 3168
>>
>> So it's pretty easy to see where the majority of the heap is being
>> used, about half the heap is being used by the ConcurrentHashMap and
>> HashMaps.
>>
>> Tracing a ConcurrentHashMap$Segment to it's root goes like this (class
>> name w/variable name):
>>
>> ConcurrentHashMap$Segment
>> ConcurrentHashMap$Segment[]
>> ConcurrentHashMap (children)
>> OutInAxisOperation
>> HashMap$Entry
>> HashMap$Entry[]
>> HashMap operationsAliasesMap
>> AxisService
>> HashMap$Entry (value)
>> HashMap$Entry[]
>> HashMap (allEndpoints)
>
> this is the place this issue was fixed.
>>
>> AxisConfiguration (axisConfiguration)
>> AxisServlet
>>
>> Hopefully that provides some insight as to where reference chain is going.
>>
>> Now, after more trial and error, I believe that I've figured out a way
>> to keep it from leaking (the service has been running overnight now
>> with a 32MB heap and it has not gone OOM yet):
>>
>> 1. Use Axis2 1.4.1 RC1
>> 2. Create a pool of stubs to use so that only one thread uses a Stub at a
>> time.
>>
>> #2 I tried after a lot of googling and mail archive searches where I
>> found a post that indicated that the creation of and use of Stubs may
>> not be thread safe. I don't know enough about the internals of axis2
>> as to why this may be an issue, but hopefully it can point the
>> developers in the right direction (still haven't been able to create a
>> small test case).
>
> then we can have a better look.
>
> thanks,
> Amila.
>>
>>
Axis 2 Memory Leak when Client is instantiated within Service - pertains to edu.emory...ConcurrentHashMap -> WAS: Re: When do cleanup functions need to be called?
Hi, I'm sorry for the re-post, but I was discussing this thread with a co-worker and we postulated that the title of the existing post (Re: When do cleanup functions need to be called?), didn't represent what this thread is actually about. I was afraid the importance of the thread might be missed, as the old title no longer applies. Hopefully, this isn't inappropriate. Sorry if I've annoyed anyone. So, renamed more appropriately, here is the thread... Many thanks, Matt Friedman. 2008/9/23 Matt Friedman <[EMAIL PROTECTED]>: > Hi David, > > Thanks very much for this. > > When you say: "Try 1.4.1. It's much better than 1.4 with this > particular memory leak.", I take it you mean it is partially fixed, > but not entirely fixed in 1.4.1. Is that correct? > > Did your pool of stubs continue to work for you? > > My goal is to completely eliminate any mem leaks from our app. > > Thanks, > Matt. > > > 2008/9/23 David Rees <[EMAIL PROTECTED]>: >> On Mon, Sep 22, 2008 at 6:39 PM, Matt Friedman <[EMAIL PROTECTED]> wrote: >>> Hi, was this issue fixed or resolved? I am running our service in >>> JProbe and we are having the identical memory leak issues as described >>> in this email chain. After roughly 900 requests the garbage collector >>> goes to 100% and out of memory occurs. >>> >>> Over half the heap is full with instances related to: >>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.* >>> which is tied to the Axis2 implementation. We don't use edu.emory.* at >>> all. >>> >>> We are running Axis2 v1.4. We're going to try a more recent release or >>> the svn nightly build. >> >> Try 1.4.1. It's much better than 1.4 with this particular memory leak. >> If your access to the stub constructor isn't synchronized, you may >> need to serialize access to stub constructors as well. >> >> -Dave >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> > > > > -- > -- Matt Friedman > -- -- Matt Friedman - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
Hi David, Thanks very much for this. When you say: "Try 1.4.1. It's much better than 1.4 with this particular memory leak.", I take it you mean it is partially fixed, but not entirely fixed in 1.4.1. Is that correct? Did your pool of stubs continue to work for you? My goal is to completely eliminate any mem leaks from our app. Thanks, Matt. 2008/9/23 David Rees <[EMAIL PROTECTED]>: > On Mon, Sep 22, 2008 at 6:39 PM, Matt Friedman <[EMAIL PROTECTED]> wrote: >> Hi, was this issue fixed or resolved? I am running our service in >> JProbe and we are having the identical memory leak issues as described >> in this email chain. After roughly 900 requests the garbage collector >> goes to 100% and out of memory occurs. >> >> Over half the heap is full with instances related to: >> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.* >> which is tied to the Axis2 implementation. We don't use edu.emory.* at >> all. >> >> We are running Axis2 v1.4. We're going to try a more recent release or >> the svn nightly build. > > Try 1.4.1. It's much better than 1.4 with this particular memory leak. > If your access to the stub constructor isn't synchronized, you may > need to serialize access to stub constructors as well. > > -Dave > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- -- Matt Friedman - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Mon, Sep 22, 2008 at 6:39 PM, Matt Friedman <[EMAIL PROTECTED]> wrote: > Hi, was this issue fixed or resolved? I am running our service in > JProbe and we are having the identical memory leak issues as described > in this email chain. After roughly 900 requests the garbage collector > goes to 100% and out of memory occurs. > > Over half the heap is full with instances related to: > edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.* > which is tied to the Axis2 implementation. We don't use edu.emory.* at > all. > > We are running Axis2 v1.4. We're going to try a more recent release or > the svn nightly build. Try 1.4.1. It's much better than 1.4 with this particular memory leak. If your access to the stub constructor isn't synchronized, you may need to serialize access to stub constructors as well. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
Hi, was this issue fixed or resolved? I am running our service in
JProbe and we are having the identical memory leak issues as described
in this email chain. After roughly 900 requests the garbage collector
goes to 100% and out of memory occurs.
Over half the heap is full with instances related to:
edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap.*
which is tied to the Axis2 implementation. We don't use edu.emory.* at
all.
We are running Axis2 v1.4. We're going to try a more recent release or
the svn nightly build.
Any thoughts or recommendations?
If you need any further information please let me know.
Many thanks,
Matt Friedman.
2008/7/31 Amila Suriarachchi <[EMAIL PROTECTED]>:
>
>
> On Wed, Jul 30, 2008 at 11:08 PM, David Rees <[EMAIL PROTECTED]> wrote:
>>
>> On Wed, Jul 30, 2008 at 4:22 AM, Amila Suriarachchi
>> <[EMAIL PROTECTED]> wrote:
>> > On Wed, Jul 30, 2008 at 2:15 PM, David Rees <[EMAIL PROTECTED]> wrote:
>> >
>> > well see this code,
>> >
>> > public static void clientCall1() {
>> > MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
>> > ClientCall1 req = new ClientCall1();
>> > ClientCall1Response res = stub.clientCall1(req);
>> > // Need to call this when calling using this function in a web service
>> > // Otherwise huge resource leak occurs
>> > stub._getServiceClient().
>> > cleanup();
>> > }
>> > }
>> >
>> > After this method call this class should be garbage collected. since
>> > stub
>> > does not have any reference after that.
>>
>> Right - but I'm not having a problem with Stubs being leaked. Here's a
>> count of various interesting classes (I've skipped some of the Java
>> core classes for brevity) of which there are a high number of
>> instances when the heap runs out of space (using a 32mb heap):
>>
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
>> 177,023, 17% heap
>>
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry[]
>> 177,023, 6% heap
>>
>> edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
>> 177,023, 8% heap
>> java.util.HashMap$Entry[] 37926, 8% heap
>> java.util.HashMap 36789, 4% heap
>> java.util.HashMap$Entry 28853, 2% heap
>> java.util.ArrayList 13698, 1% heap
>> org.apache.axis2.description.ParameterIncludeImpl 11099, 0% heap
>> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap 11064, 1%
>> heap
>> org.apache.axis2.description.PolicySubject 11063, 0% heap
>> org.apache.axis2.engine.Phase 7905, 0% heap
>> org.apache.axis2.description.AxisBindingMessage 4668, 1% heap
>> org.apache.axis2.description.AxisMessage 3168
>>
>> So it's pretty easy to see where the majority of the heap is being
>> used, about half the heap is being used by the ConcurrentHashMap and
>> HashMaps.
>>
>> Tracing a ConcurrentHashMap$Segment to it's root goes like this (class
>> name w/variable name):
>>
>> ConcurrentHashMap$Segment
>> ConcurrentHashMap$Segment[]
>> ConcurrentHashMap (children)
>> OutInAxisOperation
>> HashMap$Entry
>> HashMap$Entry[]
>> HashMap operationsAliasesMap
>> AxisService
>> HashMap$Entry (value)
>> HashMap$Entry[]
>> HashMap (allEndpoints)
>
> this is the place this issue was fixed.
>>
>> AxisConfiguration (axisConfiguration)
>> AxisServlet
>>
>> Hopefully that provides some insight as to where reference chain is going.
>>
>> Now, after more trial and error, I believe that I've figured out a way
>> to keep it from leaking (the service has been running overnight now
>> with a 32MB heap and it has not gone OOM yet):
>>
>> 1. Use Axis2 1.4.1 RC1
>> 2. Create a pool of stubs to use so that only one thread uses a Stub at a
>> time.
>>
>> #2 I tried after a lot of googling and mail archive searches where I
>> found a post that indicated that the creation of and use of Stubs may
>> not be thread safe. I don't know enough about the internals of axis2
>> as to why this may be an issue, but hopefully it can point the
>> developers in the right direction (still haven't been able to create a
>> small test case).
>
> then we can have a better look.
>
> thanks,
> Amila.
>>
>>
>> Thanks
>>
>> -Dave
>>
>> -
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>
>
>
> --
> Amila Suriarachchi,
> WSO2 Inc.
>
--
-- Matt Friedman
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Wed, Jul 30, 2008 at 11:08 PM, David Rees <[EMAIL PROTECTED]> wrote:
> On Wed, Jul 30, 2008 at 4:22 AM, Amila Suriarachchi
> <[EMAIL PROTECTED]> wrote:
> > On Wed, Jul 30, 2008 at 2:15 PM, David Rees <[EMAIL PROTECTED]> wrote:
> >
> > well see this code,
> >
> > public static void clientCall1() {
> > MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
> > ClientCall1 req = new ClientCall1();
> > ClientCall1Response res = stub.clientCall1(req);
> > // Need to call this when calling using this function in a web service
> > // Otherwise huge resource leak occurs
> > stub._getServiceClient().
> > cleanup();
> > }
> > }
> >
> > After this method call this class should be garbage collected. since stub
> > does not have any reference after that.
>
> Right - but I'm not having a problem with Stubs being leaked. Here's a
> count of various interesting classes (I've skipped some of the Java
> core classes for brevity) of which there are a high number of
> instances when the heap runs out of space (using a 32mb heap):
>
> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
> 177,023, 17% heap
>
> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry[]
> 177,023, 6% heap
>
> edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
> 177,023, 8% heap
> java.util.HashMap$Entry[] 37926, 8% heap
> java.util.HashMap 36789, 4% heap
> java.util.HashMap$Entry 28853, 2% heap
> java.util.ArrayList 13698, 1% heap
> org.apache.axis2.description.ParameterIncludeImpl 11099, 0% heap
> edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap 11064, 1%
> heap
> org.apache.axis2.description.PolicySubject 11063, 0% heap
> org.apache.axis2.engine.Phase 7905, 0% heap
> org.apache.axis2.description.AxisBindingMessage 4668, 1% heap
> org.apache.axis2.description.AxisMessage 3168
>
> So it's pretty easy to see where the majority of the heap is being
> used, about half the heap is being used by the ConcurrentHashMap and
> HashMaps.
>
> Tracing a ConcurrentHashMap$Segment to it's root goes like this (class
> name w/variable name):
>
> ConcurrentHashMap$Segment
> ConcurrentHashMap$Segment[]
> ConcurrentHashMap (children)
> OutInAxisOperation
> HashMap$Entry
> HashMap$Entry[]
> HashMap operationsAliasesMap
> AxisService
> HashMap$Entry (value)
> HashMap$Entry[]
> HashMap (allEndpoints)
this is the place this issue was fixed.
>
> AxisConfiguration (axisConfiguration)
> AxisServlet
>
> Hopefully that provides some insight as to where reference chain is going.
>
> Now, after more trial and error, I believe that I've figured out a way
> to keep it from leaking (the service has been running overnight now
> with a 32MB heap and it has not gone OOM yet):
>
> 1. Use Axis2 1.4.1 RC1
> 2. Create a pool of stubs to use so that only one thread uses a Stub at a
> time.
>
> #2 I tried after a lot of googling and mail archive searches where I
> found a post that indicated that the creation of and use of Stubs may
> not be thread safe. I don't know enough about the internals of axis2
> as to why this may be an issue, but hopefully it can point the
> developers in the right direction (still haven't been able to create a
> small test case).
then we can have a better look.
thanks,
Amila.
>
>
> Thanks
>
> -Dave
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Amila Suriarachchi,
WSO2 Inc.
Re: When do cleanup functions need to be called?
On Wed, Jul 30, 2008 at 2:09 AM, Pierre Muller <[EMAIL PROTECTED]> wrote: > On spec, wouldn't it be related to the scope of your services ? > http://www.developer.com/java/web/article.php/3620661 I don't think so - The default scope (request) should be fine. I don't care about anything longer than that. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Wed, Jul 30, 2008 at 4:22 AM, Amila Suriarachchi
<[EMAIL PROTECTED]> wrote:
> On Wed, Jul 30, 2008 at 2:15 PM, David Rees <[EMAIL PROTECTED]> wrote:
>
> well see this code,
>
> public static void clientCall1() {
> MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
> ClientCall1 req = new ClientCall1();
> ClientCall1Response res = stub.clientCall1(req);
> // Need to call this when calling using this function in a web service
> // Otherwise huge resource leak occurs
> stub._getServiceClient().
> cleanup();
> }
> }
>
> After this method call this class should be garbage collected. since stub
> does not have any reference after that.
Right - but I'm not having a problem with Stubs being leaked. Here's a
count of various interesting classes (I've skipped some of the Java
core classes for brevity) of which there are a high number of
instances when the heap runs out of space (using a 32mb heap):
edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
177,023, 17% heap
edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry[]
177,023, 6% heap
edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
177,023, 8% heap
java.util.HashMap$Entry[] 37926, 8% heap
java.util.HashMap 36789, 4% heap
java.util.HashMap$Entry 28853, 2% heap
java.util.ArrayList 13698, 1% heap
org.apache.axis2.description.ParameterIncludeImpl 11099, 0% heap
edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap 11064, 1% heap
org.apache.axis2.description.PolicySubject 11063, 0% heap
org.apache.axis2.engine.Phase 7905, 0% heap
org.apache.axis2.description.AxisBindingMessage 4668, 1% heap
org.apache.axis2.description.AxisMessage 3168
So it's pretty easy to see where the majority of the heap is being
used, about half the heap is being used by the ConcurrentHashMap and
HashMaps.
Tracing a ConcurrentHashMap$Segment to it's root goes like this (class
name w/variable name):
ConcurrentHashMap$Segment
ConcurrentHashMap$Segment[]
ConcurrentHashMap (children)
OutInAxisOperation
HashMap$Entry
HashMap$Entry[]
HashMap operationsAliasesMap
AxisService
HashMap$Entry (value)
HashMap$Entry[]
HashMap (allEndpoints)
AxisConfiguration (axisConfiguration)
AxisServlet
Hopefully that provides some insight as to where reference chain is going.
Now, after more trial and error, I believe that I've figured out a way
to keep it from leaking (the service has been running overnight now
with a 32MB heap and it has not gone OOM yet):
1. Use Axis2 1.4.1 RC1
2. Create a pool of stubs to use so that only one thread uses a Stub at a time.
#2 I tried after a lot of googling and mail archive searches where I
found a post that indicated that the creation of and use of Stubs may
not be thread safe. I don't know enough about the internals of axis2
as to why this may be an issue, but hopefully it can point the
developers in the right direction (still haven't been able to create a
small test case).
Thanks
-Dave
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Wed, Jul 30, 2008 at 2:15 PM, David Rees <[EMAIL PROTECTED]> wrote:
> On Tue, Jul 29, 2008 at 10:52 PM, Amila Suriarachchi
> <[EMAIL PROTECTED]> wrote:
> > this method should be call when the MyServiceStub (see finalize method in
> > ServiceClient) is garbage collected. I am not sure why it is not being
> > called automatically.
>
> I don't think that it's being garbage collected at all. From what I
> can tell, a bunch of objects are still being referenced by the
> AxisConfiguration referenced by the AxisServlet.
well see this code,
public static void clientCall1() {
MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
ClientCall1 req = new ClientCall1();
ClientCall1Response res = stub.clientCall1(req);
// Need to call this when calling using this function in a web service
// Otherwise huge resource leak occurs
stub._getServiceClient().cleanup();
}
}
After this method call this class should be garbage collected. since stub
does not have any reference after that.
>
>
> > In RC1 we fixed a memory leak regarding this. But I can not realy think
> the
> > reason for your problem.
>
> Yes, 1.4.1 RC1 makes the situation a lot better than 1.4 - Calling
> cleanup has no effect on the memory leak (see my earlier followup
> email).
>
> > you can get the AxisConfiguration object like this,
> > AxisConfiguration axiConfiguration =
> >
> MessageContext.getCurrentMessageContext().getConfigurationContext().getAxisConfiguration();
> >
> > and check whether there are some additional objects there after calling
> > clean up.
> >
> > Please log a jira.
>
> Once I can create a small reproducable test-case I'll log a jira issue.
yep. this helps a lot.
thanks,
Amila.
>
>
> Let me know if there is any other information I can gather or if
> there's anything else you want me to try in the meantime.
>
> Thanks,
>
> -Dave
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Amila Suriarachchi,
WSO2 Inc.
Re: When do cleanup functions need to be called?
On spec, wouldn't it be related to the scope of your services ? http://www.developer.com/java/web/article.php/3620661 David Rees a écrit : On Tue, Jul 29, 2008 at 10:52 PM, Amila Suriarachchi <[EMAIL PROTECTED]> wrote: this method should be call when the MyServiceStub (see finalize method in ServiceClient) is garbage collected. I am not sure why it is not being called automatically. I don't think that it's being garbage collected at all. From what I can tell, a bunch of objects are still being referenced by the AxisConfiguration referenced by the AxisServlet. In RC1 we fixed a memory leak regarding this. But I can not realy think the reason for your problem. Yes, 1.4.1 RC1 makes the situation a lot better than 1.4 - Calling cleanup has no effect on the memory leak (see my earlier followup email). you can get the AxisConfiguration object like this, AxisConfiguration axiConfiguration = MessageContext.getCurrentMessageContext().getConfigurationContext().getAxisConfiguration(); and check whether there are some additional objects there after calling clean up. Please log a jira. Once I can create a small reproducable test-case I'll log a jira issue. Let me know if there is any other information I can gather or if there's anything else you want me to try in the meantime. Thanks, -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ** Si vous n'êtes pas le destinataire désigné de ce message ou une personne autorisée à l'utiliser, toute distribution, copie, publication ou usage à quelques fins que ce soit des informations dans ce message sont interdits. Merci d'informer immédiatement l'expéditeur par messagerie, et, de détruire ce message. This e-mail is confidential. If you are not the addressee or an authorized recipient of this message, any distribution, copying, publication or use of this information for any purpose is prohibited. Please notify the sender immediately by e-mail and then delete this message. ** - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Tue, Jul 29, 2008 at 10:52 PM, Amila Suriarachchi <[EMAIL PROTECTED]> wrote: > this method should be call when the MyServiceStub (see finalize method in > ServiceClient) is garbage collected. I am not sure why it is not being > called automatically. I don't think that it's being garbage collected at all. From what I can tell, a bunch of objects are still being referenced by the AxisConfiguration referenced by the AxisServlet. > In RC1 we fixed a memory leak regarding this. But I can not realy think the > reason for your problem. Yes, 1.4.1 RC1 makes the situation a lot better than 1.4 - Calling cleanup has no effect on the memory leak (see my earlier followup email). > you can get the AxisConfiguration object like this, > AxisConfiguration axiConfiguration = > MessageContext.getCurrentMessageContext().getConfigurationContext().getAxisConfiguration(); > > and check whether there are some additional objects there after calling > clean up. > > Please log a jira. Once I can create a small reproducable test-case I'll log a jira issue. Let me know if there is any other information I can gather or if there's anything else you want me to try in the meantime. Thanks, -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: When do cleanup functions need to be called?
On Wed, Jul 30, 2008 at 6:21 AM, David Rees <[EMAIL PROTECTED]> wrote:
> I'm developing an application using Axis2 1.4 which consists of a
> client and service. Stubs are generated using the wsdl2java tool using
> ADB.
>
> I am seeing what appears to be a huge resource leak when using the
> client code from the service. After processing a couple dozen service
> requests (each service requests runs a few client requests), the heap
> quickly fills up and the JVM runs out of memory.
>
> Analysing the heap shows hundreds of thousands of
> ConcurrentHashMap$Segment, ConcurrentHashMap$HashEntry[] and
> ReentrantLock$NonfairSync objects, and tens of thousands of HashMap,
> HashMap$Entry[], String, Object[], ArrayList,
> org.apache.axis2.description.ParameterIncludeImpl,
> org.apache.axis2.description.PolicySubject for example.
>
> Executing the service without making any client calls works just fine.
>
> Executing the client by itself in a standalone app works just fine.
>
> After pulling my hair out for the past 24 hours trying to figure out
> the root cause and desperately googling for answers, I finally found
> that calling MyStub._getService().cleanup() appears to fix the leak
> (at least the service can process requests for over an hour where it
> would die within minutes).
>
> So, my questions are:
>
> Is it normal to have to call ServiceClient.cleanup()?
> Why do I have to explicitly call it when using the client code from my
> service and not have to call it when running the client code as a
> standalone application?
> Should I be reusing stubs if possible?
>
> Code looks like this:
>
> public class MyService implements MyServiceSkeletonInterface {
> public MyProcessResponse myProcess() {
> MyProcessResponse res = new MyProcessResponse();
> WebServiceUtil.clientCall1();
> WebServiceUtil.clientCall2();
> res.setProcessResponseResult("OK");
> return res;
> }
> }
>
> public class WebServiceUtil {
> public static void clientCall1() {
> MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
> ClientCall1 req = new ClientCall1();
> ClientCall1Response res = stub.clientCall1(req);
> // Need to call this when calling using this function in a web service
> // Otherwise huge resource leak occurs
> stub._getServiceClient().cleanup();
this method should be call when the MyServiceStub (see finalize method in
ServiceClient) is garbage collected. I am not sure why it is not being
called automatically.
In RC1 we fixed a memory leak regarding this. But I can not realy think the
reason for your problem.
you can get the AxisConfiguration object like this,
AxisConfiguration axiConfiguration =
MessageContext.getCurrentMessageContext().getConfigurationContext().getAxisConfiguration();
and check whether there are some additional objects there after calling
clean up.
Please log a jira.
thanks,
Amila.
> }
> }
>
> Any thoughts appreciated.
>
> -Dave
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Amila Suriarachchi,
WSO2 Inc.
Re: When do cleanup functions need to be called?
On Tue, Jul 29, 2008 at 5:51 PM, David Rees <[EMAIL PROTECTED]> wrote: > After pulling my hair out for the past 24 hours trying to figure out > the root cause and desperately googling for answers, I finally found > that calling MyStub._getService().cleanup() appears to fix the leak > (at least the service can process requests for over an hour where it > would die within minutes). Unfortunately, I was mistaken. Calling cleanup() doesn't appear to help at all, unfortunately I also switched to Axis2 1.4.1 RC1 at the same time which appears to to have reduced the rate at which the heap bloats, but eventually, it still suffers the same fate. However, eventually, the service (running under Tomcat 5.5.26) still runs out of heap space with the same symptoms - hundreds of thousands of ConcurrentHashMap objects. These all appear to trace back to the AxisServlet through this chain: AxisBindingOperation -> AxisBindingMessage ->ConcurrentHashMap -> AxisBinding -> AxisBindingOperation -> AxisEndpoint -> HashMap -> AxisService -> OutInAxisOperation -> HashMap -> AxisConfiguration -> AxisServlet. So what's going on here? I can't imagine that I'm the only one making Axis2 client calls from a Axis2 web service and running into this issue - either I'm doing something unique or I'm doing something wrong... Thanks Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
When do cleanup functions need to be called?
I'm developing an application using Axis2 1.4 which consists of a
client and service. Stubs are generated using the wsdl2java tool using
ADB.
I am seeing what appears to be a huge resource leak when using the
client code from the service. After processing a couple dozen service
requests (each service requests runs a few client requests), the heap
quickly fills up and the JVM runs out of memory.
Analysing the heap shows hundreds of thousands of
ConcurrentHashMap$Segment, ConcurrentHashMap$HashEntry[] and
ReentrantLock$NonfairSync objects, and tens of thousands of HashMap,
HashMap$Entry[], String, Object[], ArrayList,
org.apache.axis2.description.ParameterIncludeImpl,
org.apache.axis2.description.PolicySubject for example.
Executing the service without making any client calls works just fine.
Executing the client by itself in a standalone app works just fine.
After pulling my hair out for the past 24 hours trying to figure out
the root cause and desperately googling for answers, I finally found
that calling MyStub._getService().cleanup() appears to fix the leak
(at least the service can process requests for over an hour where it
would die within minutes).
So, my questions are:
Is it normal to have to call ServiceClient.cleanup()?
Why do I have to explicitly call it when using the client code from my
service and not have to call it when running the client code as a
standalone application?
Should I be reusing stubs if possible?
Code looks like this:
public class MyService implements MyServiceSkeletonInterface {
public MyProcessResponse myProcess() {
MyProcessResponse res = new MyProcessResponse();
WebServiceUtil.clientCall1();
WebServiceUtil.clientCall2();
res.setProcessResponseResult("OK");
return res;
}
}
public class WebServiceUtil {
public static void clientCall1() {
MyServiceStub stub = new MyServiceStub("http://example.com/myservice";);
ClientCall1 req = new ClientCall1();
ClientCall1Response res = stub.clientCall1(req);
// Need to call this when calling using this function in a web service
// Otherwise huge resource leak occurs
stub._getServiceClient().cleanup();
}
}
Any thoughts appreciated.
-Dave
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
