On Thu, May 19, 2011 at 20:09, Hiranya Jayathilaka <[email protected]> wrote: > Synapse creates an Axis2MessageContext for each message received. Each > Axis2MessageContext instance encapsulates a MessageContext object. But > Synapse does not keep them in memory for too long. As soon as messages are > mediated and sent to the corresponding endpoints/clients, they are > discarded.
Not sure that this is entirely correct. In a request-response exchange, doesn't the callback keep a reference to the outgoing message context? What I've seen in the past is that some configuration problems may cause Synapse to believe that a message exchange is request-response while the outgoing transport is actually doing out-only. In that case, the callback handlers just continue to pile up in memory until they expire. IIRC, at the time I observed this, the default timeout was 24h. Abid's observation that this only occurs for failures may point to the same root cause, i.e. callbacks not being removed. > These objects could be fairly big since they contain message > payload, properties and all other context information of messages. So if > Synapse is keeping them in memory forever, we won't be able to run any > scenario with Synapse for more than several hours. But as Paul mentioned we > have seen some pretty heavy duty deployments of Synapse dealing with > hundreds of transactions every second. One of the key strengths of Synapse > is that it can handle very high volumes of messages with a very small memory > footprint. > > How much memory have you allocated for JBoss server? Also what is your > Synapse configuration like? What's the memory usage pattern when you run > Synapse standalone (ie without JBoss)? > > Thanks, > Hiranya > > On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected]>wrote: > >> Hello Paul, >> >> What you are telling me is a good news. The facts I have here, are >> different. I ran the server for a load-test and ran only 26000 message (of >> about 1K each message size), that made the GC threads taking over every >> thing, the java process was so stuck that the jboss-shutdown command could >> not bring it down. >> >> And my trace log (memory profiling) showed that one instance of both >> MessageContext and Axis2MessageContext were leaked for every message that is >> delivered/processed. >> >> I am using 1.2 GA of synapse. And I am running it in JBoss. >> >> And will be more than happy to send the thread dump, but that is very big. >> >> Once again thanks for getting back to me and I am looking for good pointers >> from you very soon. >> >> Regards, >> Abid >> >> >> >> >> -----Original Message----- >> From: Paul Fremantle [mailto:[email protected]] >> Sent: Wednesday, May 18, 2011 8:39 PM >> To: [email protected] >> Subject: Re: Please help: does Synapse leak memory? >> >> Abid >> >> Can you give us some idea of the flows? I know that Synapse is usually >> memory safe - I have thread dumps from people who have pumped 134 >> terabytes of data, 1.4 billion messages and all this with a max heap >> size of <2Gb and no leaks. >> >> Paul >> >> On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <[email protected]> >> wrote: >> > Hello all, >> > >> > I am troubleshooting an instance where I had used Synapse, the issue is >> that after delivering few thousand message server runs out of memory. >> > >> > I investigated and found out that for each message that is delivered >> Synapse is leaking following two objects: >> > >> > 1) org.apache.axis2.context.MessageContext >> > >> > 2) org.apache.synapse.core.axis2.Axis2MessageContext >> > >> > I thought that it might be related to threads and ThreadLocal, but when I >> scanned the thread local of all 160+ threads those objects were not there. >> > >> > It seems that synapse is keeping those two object in its some internal >> object, that is difficult to find. >> > >> > Because of this problem those objects; server leaks memory at very high >> rate, and freezes after few hours of running? >> > >> > Any immediate response will be appreciated, >> > >> > >> > Thanks, >> > Abid >> > >> >> >> >> -- >> Paul Fremantle >> Co-Founder and CTO, WSO2 >> Apache Synapse PMC Chair >> OASIS WS-RX TC Co-chair >> >> blog: http://pzf.fremantle.org >> [email protected] >> >> "Oxygenating the Web Service Platform", www.wso2.com >> > > > > -- > Hiranya Jayathilaka > Senior Software Engineer; > WSO2 Inc.; http://wso2.org > E-mail: [email protected]; Mobile: +94 77 633 3491 > Blog: http://techfeast-hiranya.blogspot.com >
