Hi eric, thx for all your help, maybe smtp-source ( which is included in postfix ) can help you with stress testing.
Bye Norman 2010/3/31, Eric Charles <[email protected]>: > Hi Norman, > > There are not so much .m64 files, so even it it shows that a mail > processing didn't reach the end and may have cause a leak, I don't think > it would make crash my process so quick (between 10 and 24 hours). > > I quickly analysed a few dumps last week : there is always a class that > took 30/40% of the memory. Sometimes a activemq class, sometimes not > (don't remember the details). > Also notable, the stack trace (the place where the OOM gives problems) > varies. > Today stack was: > org.apache.camel.RuntimeCamelException: > org.apache.camel.CamelExchangeException: Error processing Exchange. > Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102, > responseRequired = true, messageId = > ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null, > originalTransactionId = null, producerId = > ID:srv001-51032-1270011735798-2:0:24:1, destination = > queue://processor.root, transactionId = null, expiration = 0, timestamp > = 1270014325685, arrival = 0, brokerInTime = 1270014326893, > brokerOutTime = 1270014333023, correlationId = null, replyTo = null, > persistent = true, type = null, priority = 4, groupID = null, > groupSequence = 0, targetConsumerId = null, compressed = false, userID = > null, content = org.apache.activemq.util.byteseque...@69b568d0, > marshalledProperties = null, dataStructure = null, redeliveryCounter = > 0, size = 10718, properties = null, readOnlyProperties = true, > readOnlyBody = true, droppable = false}]. Caused by: > [java.lang.OutOfMemoryError - Java heap space] > at > org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) > at > org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) > ... > > > > I also think the tmp files are not the main cause. > > I have now james running in eclipse and I made already made a little > trip in the code. Quite impressive since last time I looked at it. > I will try to stress james with a small smtp/pop3 client (I used > postage, but a simple class with commons-net behind) may be easier and > see what happens (eventually with eclipse profiler). > > It will be for this weekend for me. > > Bye, > > Eric > > > On 03/31/2010 07:11 PM, Norman Maurer wrote: >> Hi Eric, >> >> thx for keeping us in the loop... I'm still not sure why the .m64 >> files are still in the tmp folder sometimes.. But I suspect the files >> are not the cause of the OOM. >> >> I didn't get a OOM since yesterday morning (the time I deployed >> current trunk version), but just found to new .m64 files.. >> >> So I'm still searching for the real cause. If nothing helps I will >> need to use a profiler to find th leak. >> >> Bye, >> Norman >> >> 2010/3/31 Eric Charles<[email protected]>: >> >>> Hi Norman, >>> >>> I had defined the Null mailet >>> >>> <mailet match="HostIsLocal" class="Null"> >>> <processor> local-address-error</processor> >>> <notice>550 - Requested action not taken: no such user here</notice> >>> </mailet> >>> >>> but now, I use the standard config >>> >>> <mailet match="HostIsLocal" class="ToProcessor"> >>> <processor> local-address-error</processor> >>> <notice>550 - Requested action not taken: no such user here</notice> >>> </mailet> >>> >>> I still have the OOM. >>> >>> I will now deploy a fresh svn with your last commits and enable the >>> ValidRcptHandler. >>> I keep you posted with the result, >>> >>> Tks, >>> Eric >>> >>> >>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>> >>>> Hi Eric, >>>> >>>> you said all the files are related to address-errors , could you show >>>> me your address error processor config? >>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>> the smtpserver.xml file? >>>> >>>> Thx >>>> Norman >>>> >>>> 2010/3/30, Eric Charles<[email protected]>: >>>> >>>> >>>>> Oops, no, the files are still there (only [email protected]). >>>>> Eric >>>>> >>>>> >>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>> >>>>> >>>>>> Hi Norman, >>>>>> >>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>> file, on the second after, it is no more there, so the dispose works >>>>>> as it should). >>>>>> >>>>>> I keep you posted with our eventual future OOM. >>>>>> >>>>>> Tks, >>>>>> Eric >>>>>> >>>>>> >>>>>> >>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>> >>>>>> >>>>>>> Hi Norman, >>>>>>> >>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so for >>>>>>> <[email protected]>). >>>>>>> They are from various size (with and without attachment). >>>>>>> They are well formed (I can open the downloaded file with >>>>>>> thunderbird) >>>>>>> >>>>>>> However, when I sent a mail to [email protected], I don't see it in >>>>>>> the /tmp >>>>>>> >>>>>>> Tks, >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>> >>>>>>> >>>>>>>> Hi Eric, >>>>>>>> >>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>> could >>>>>>>> you check somehow if the .m64 files are files which are related to >>>>>>>> successfully delivered mail ? >>>>>>>> >>>>>>>> Thx, >>>>>>>> Norman >>>>>>>> >>>>>>>> >>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>> >>>>>>>> >>>>>>>>> Norman, >>>>>>>>> >>>>>>>>> Done. Now, we have to wait... >>>>>>>>> >>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>> >>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump and >>>>>>>>> analyse >>>>>>>>> it. >>>>>>>>> At the beginning, everything seems ok, and often, when I come back, >>>>>>>>> the OOM >>>>>>>>> has already occured. >>>>>>>>> >>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception (if >>>>>>>>> any, >>>>>>>>> let's hope not). >>>>>>>>> >>>>>>>>> The *.m64 are a clue and we should ensure that they are removed in >>>>>>>>> all >>>>>>>>> circumstances. >>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>> files). >>>>>>>>> We should also ensure after that there are no other causes to the >>>>>>>>> leaks. >>>>>>>>> >>>>>>>>> Tks, >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully >>>>>>>>>> this >>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>> >>>>>>>>>> Thx, >>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2010/3/29 Norman Maurer<[email protected]>: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Hi Eric, >>>>>>>>>>> >>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>> (The >>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>> thats >>>>>>>>>>> a >>>>>>>>>>> good pointer. I will try to debug the problem later or tomorrow. >>>>>>>>>>> But I >>>>>>>>>>> suspect you are right about the tmp files and jms producers.. Did >>>>>>>>>>> you >>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>> >>>>>>>>>>> Thx, >>>>>>>>>>> Norman >>>>>>>>>>> >>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>> >>>>>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>> OutOfMemory >>>>>>>>>>>> Exception. >>>>>>>>>>>> >>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and >>>>>>>>>>>> always >>>>>>>>>>>> found >>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but nothing >>>>>>>>>>>> helped. >>>>>>>>>>>> >>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by >>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, sometimes >>>>>>>>>>>> by a >>>>>>>>>>>> activemq class,... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >>>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
