Hi Sathwik, Try wrapping the process in a Repeat Until activity with an always false condition (e.g., true() = false() ). I usually put a <wait><for /></wait> so that the service gets called about every 30 seconds (back-off). Start 1, 000 instances of that process (effectively making 2, 000 req/min) and let them run for a while. The memory will soon start leaking.
ODE already has a workaround for this particular memory issue on process termination. The key is to make the process run for a long time and make lots of calls to the same service from within the lifetime of one process. I'll work on extracting the relevant project artifacts and will send separately as a ZIP. Thanks for looking into this. On 25 July 2014 08:24, Sathwik B P <[email protected]> wrote: > Hi Stan, > > Would like your help in replicating this issue. > > I have devised a process which calls an external service. I have a > continuous looping call to the process from the soapUI. > I have run about 4000 instances a couple of times, and I don't see > AxisService Instances going beyond 110 as my threads per host is equal to > 100. > > regards, > sathwik > > > > > On Tue, Jul 15, 2014 at 12:59 PM, Sathwik B P <[email protected]> > wrote: > > > Hi Stan, > > > > Thanks for reporting and your patch. > > > > Will look into it when I get some freetime. > > > > For your question about building ODE with axis2 1.6, it's not possible at > > the moment as it needs some changes in the Axiom usage within ODE. > > > > Will see if we can bring it on 1.3.7 release, though the timelines are > not > > fixed yet. > > > > regards, > > sathwik > > > > > > On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <[email protected]> > > wrote: > > > >> For those interested, I ended up patching Axis2 1.5.6 from source with > >> > >> > https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch > >> The patch has been adapted from two commits on the 1_6 branch. I > >> re-compiled the kernel module and swapped the one that comes with the > >> binary distribution of ODE. This seems to have resolved the issue for > me. > >> > >> I also found https://issues.apache.org/jira/browse/ODE-585, however > it's > >> tagged for 2.0. IMHO memory leaks are a pretty major issue and should > get > >> addressed in whatever is the current stable release? > >> > >> > >> On 14 July 2014 14:31, Stan Angeloff <[email protected]> wrote: > >> > >> > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about > >> > building ODE and bundling it with Axis 1.6? > >> > > >> > Digging a bit deeper in the log files with > >> > log4j.category.org.apache.axis2=DEBUG and > >> > log4j.category.org.apache.ode=DEBUG configured yields interesting > >> results. > >> > When a ServiceClient instance is created, the second parameter is > NULL. > >> > When this is the case, Axis2 will create a new anonymous service ( > >> > > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137 > >> ). > >> > The anonymous service is then added to the AxisConfiguration instance > ( > >> > > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176 > >> > ). > >> > > >> > Internally, this creates a new AxisServiceGroup ( > >> > > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315 > >> ). > >> > This then adds the new anonymous service to an allEndpoints property ( > >> > > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373 > >> > ). > >> > > >> > When ODE calls setAxisService, Axis2 clean up the service ( > >> > > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612 > >> ). > >> > It does not, however, clean up the reference from allEndpoints. > >> > > >> > The filtered catalina.out > >> > < > >> > https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered > >> > > >> > log is attached for reference. Analysing the heap dumps indicates that > >> > allEndpoints is indeed the culprit. > >> > > >> > Axis 1.6 seems to have a fix for this issue: > >> > > >> > > >> > https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679 > >> > > >> > Is it possible to use Axis 1.6 in ODE? How does one go about building > >> ODE > >> > and bundling it with Axis 1.6? > >> > Should the ODE 1.3.x branch be updated to use the latest stable Axis2? > >> > > >> > Regards, > >> > Stan > >> > > >> > > >> > > >> > > >> > On 14 July 2014 13:45, Stan Angeloff <[email protected]> wrote: > >> > > >> >> Hello, > >> >> > >> >> I've been playing with ODE for quite a while now and one of my most > >> >> recent attempts tried to run 5, 000 simultaneous instances of one > >> process. > >> >> Unfortunately after a while ODE uses up all of the allowed memory and > >> barfs > >> >> 'Java heap space' continuously in the log. > >> >> > >> >> My set up is very basic, starting from > >> >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an > >> >> excellent learning tool). The main .bpel file consists of a > continuous > >> >> Repeat Until inside of which there are two sequential invoke > activities > >> >> calling external SOAP services. > >> >> > >> >> When the 5, 000 instances are started, the memory footprint is fairly > >> >> stable. After a while, though, > >> org.apache.axis2.engine.AxisConfiguration > >> >> accumulates a lot of org.apache.axis2.description.AxisService > instances > >> >> (all of which have their name property set to 'anonService' + hash). > >> >> Those keep growing until eventually the server stops responding due > to > >> >> exhausted memory. I have taken heap dumps at various stages and > >> examined > >> >> those. Eclipse was even smart enough to detect the potential memory > >> leak on > >> >> the latest dump just as the server crashed. > >> >> > >> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by > >> >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies > >> >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one > instance > >> of > >> >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by > "<system > >> class > >> >>> loader>". > >> >> > >> >> > >> >> I'll be happy to export the complete project, note however it doesn't > >> >> matter what services are invoked or what the .bpel does so long it's > a) > >> >> continuous, e.g., does not stop b) calls an external service. > >> >> > >> >> I tried looking into ODE/Axis2 to try and figure out what might be > >> going > >> >> wrong. My understanding is that service clients are cached and > recycled > >> >> in SoapExternalService.java ( > >> >> > >> > https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265 > >> ) > >> >> When a new ServiceClient instance is created, it calls the > >> >> configureServiceClient method within Axis2 with axisService = null ( > >> >> > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143 > >> ). > >> >> That in turn calls createAnonymousService. Later on, ODE calls > >> >> setAxisService which forces Axis2 to dispose of the anonymous service > >> it > >> >> created earlier. > >> >> > >> >> If the above is indeed what is happening, then there should be no > >> memory > >> >> leaks. However, in the memory dumps the number of instances of > >> >> org.apache.axis2.description.AxisService keeps growing. I've tried > >> >> increasing the maximum heap size up to 8GB and it will fail again, it > >> is > >> >> just a matter of time. > >> >> > >> >> Has anyone else experienced this? Is anyone interested in trying to > >> help > >> >> me narrow down the cause of this? Is it possible it's not even a > memory > >> >> issue? > >> >> > >> >> Regards, > >> >> Stan > >> >> > >> > > >> > > >> > > > > >
