For those interested, I ended up patching Axis2 1.5.6 from source with https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch The patch has been adapted from two commits on the 1_6 branch. I re-compiled the kernel module and swapped the one that comes with the binary distribution of ODE. This seems to have resolved the issue for me.
I also found https://issues.apache.org/jira/browse/ODE-585, however it's tagged for 2.0. IMHO memory leaks are a pretty major issue and should get addressed in whatever is the current stable release? On 14 July 2014 14:31, Stan Angeloff <[email protected]> wrote: > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about > building ODE and bundling it with Axis 1.6? > > Digging a bit deeper in the log files with > log4j.category.org.apache.axis2=DEBUG and > log4j.category.org.apache.ode=DEBUG configured yields interesting results. > When a ServiceClient instance is created, the second parameter is NULL. > When this is the case, Axis2 will create a new anonymous service ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137). > The anonymous service is then added to the AxisConfiguration instance ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176 > ). > > Internally, this creates a new AxisServiceGroup ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315). > This then adds the new anonymous service to an allEndpoints property ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373 > ). > > When ODE calls setAxisService, Axis2 clean up the service ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612). > It does not, however, clean up the reference from allEndpoints. > > The filtered catalina.out > <https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered> > log is attached for reference. Analysing the heap dumps indicates that > allEndpoints is indeed the culprit. > > Axis 1.6 seems to have a fix for this issue: > > https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679 > > Is it possible to use Axis 1.6 in ODE? How does one go about building ODE > and bundling it with Axis 1.6? > Should the ODE 1.3.x branch be updated to use the latest stable Axis2? > > Regards, > Stan > > > > > On 14 July 2014 13:45, Stan Angeloff <[email protected]> wrote: > >> Hello, >> >> I've been playing with ODE for quite a while now and one of my most >> recent attempts tried to run 5, 000 simultaneous instances of one process. >> Unfortunately after a while ODE uses up all of the allowed memory and barfs >> 'Java heap space' continuously in the log. >> >> My set up is very basic, starting from >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an >> excellent learning tool). The main .bpel file consists of a continuous >> Repeat Until inside of which there are two sequential invoke activities >> calling external SOAP services. >> >> When the 5, 000 instances are started, the memory footprint is fairly >> stable. After a while, though, org.apache.axis2.engine.AxisConfiguration >> accumulates a lot of org.apache.axis2.description.AxisService instances >> (all of which have their name property set to 'anonService' + hash). >> Those keep growing until eventually the server stops responding due to >> exhausted memory. I have taken heap dumps at various stages and examined >> those. Eclipse was even smart enough to detect the potential memory leak on >> the latest dump just as the server crashed. >> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class >>> loader>". >> >> >> I'll be happy to export the complete project, note however it doesn't >> matter what services are invoked or what the .bpel does so long it's a) >> continuous, e.g., does not stop b) calls an external service. >> >> I tried looking into ODE/Axis2 to try and figure out what might be going >> wrong. My understanding is that service clients are cached and recycled >> in SoapExternalService.java ( >> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265) >> When a new ServiceClient instance is created, it calls the >> configureServiceClient method within Axis2 with axisService = null ( >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143). >> That in turn calls createAnonymousService. Later on, ODE calls >> setAxisService which forces Axis2 to dispose of the anonymous service it >> created earlier. >> >> If the above is indeed what is happening, then there should be no memory >> leaks. However, in the memory dumps the number of instances of >> org.apache.axis2.description.AxisService keeps growing. I've tried >> increasing the maximum heap size up to 8GB and it will fail again, it is >> just a matter of time. >> >> Has anyone else experienced this? Is anyone interested in trying to help >> me narrow down the cause of this? Is it possible it's not even a memory >> issue? >> >> Regards, >> Stan >> > >
