Hi Eric, thx again for your tests...
So I think we have two independent problems here: 1) The .m64 files which are not deleted always: My guess here is that there is something wrong with processing Mail objects with state Null. I will investigate futher.. I also found a possible problem with .m64 files which get not deleted when a Exception is throws while processing the camel routes. I will patch this soon. 2) The OOM My guess here is that ActiveMQ is not freeing up memory. Not sure why yet I think 2) is the more criticial problem and should get higher priority to get fixed.. Bye, Norman 2010/4/1 Eric Charles <[email protected]>: > Hi Norman, > > Tks for the smpt-source pointer. > I looked at it, but finally, I think I will develop those little classes to > have more control on it. > I will put it on github. > > I launched latest trunk james yesterday evening. > During the night, 3 hours after the launch, the following occured: > > KahDB slower and slower... > > INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow > KahaDB access: cleanup took 2330 > INFO 23:38:04,591 | james.mailetcontext | Error while storing mail. > org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be > parsed.; > nested exception is: > org.apache.james.imap.mailbox.StorageException: failed. Transaction > commit failed.; > nested exception is: > <openjpa-1.2.1-r752877:753278 fatal store error> > org.apache.openjpa.persistence.RollbackException: The transaction has been > rolled back. See the nested exceptions for details on the errors that > occurred. > at > org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) > > at > org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130) > ... 55 more > Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java > exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt > 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets, > mediaType, subType, textu > alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260, > (byte[]) [...@7a00e70c, (long) 47887, (String) text, (String) html, (long) > 876]} [code=0, state=XJ001] > at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192) > at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57) > at > org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866) > at > org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269) > > So still a OOM exception that was shown by yet-another-component (in this > case, the StoreMailbox). > > > There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its > job). > All 4 files were 0 bytes. > > I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max > memory). > > With the previous parameters ( -Xmx512m), the process was taking the whole > 512MB. > > Tks, > Eric > > > On 03/31/2010 08:26 PM, Norman Maurer wrote: >> >> Hi eric, >> >> thx for all your help, maybe smtp-source ( which is included in >> postfix ) can help you with stress testing. >> >> Bye >> Norman >> >> 2010/3/31, Eric Charles<[email protected]>: >> >>> >>> Hi Norman, >>> >>> There are not so much .m64 files, so even it it shows that a mail >>> processing didn't reach the end and may have cause a leak, I don't think >>> it would make crash my process so quick (between 10 and 24 hours). >>> >>> I quickly analysed a few dumps last week : there is always a class that >>> took 30/40% of the memory. Sometimes a activemq class, sometimes not >>> (don't remember the details). >>> Also notable, the stack trace (the place where the OOM gives problems) >>> varies. >>> Today stack was: >>> org.apache.camel.RuntimeCamelException: >>> org.apache.camel.CamelExchangeException: Error processing Exchange. >>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102, >>> responseRequired = true, messageId = >>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null, >>> originalTransactionId = null, producerId = >>> ID:srv001-51032-1270011735798-2:0:24:1, destination = >>> queue://processor.root, transactionId = null, expiration = 0, timestamp >>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893, >>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null, >>> persistent = true, type = null, priority = 4, groupID = null, >>> groupSequence = 0, targetConsumerId = null, compressed = false, userID = >>> null, content = org.apache.activemq.util.byteseque...@69b568d0, >>> marshalledProperties = null, dataStructure = null, redeliveryCounter = >>> 0, size = 10718, properties = null, readOnlyProperties = true, >>> readOnlyBody = true, droppable = false}]. Caused by: >>> [java.lang.OutOfMemoryError - Java heap space] >>> at >>> >>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >>> at >>> >>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >>> ... >>> >>> >>> >>> I also think the tmp files are not the main cause. >>> >>> I have now james running in eclipse and I made already made a little >>> trip in the code. Quite impressive since last time I looked at it. >>> I will try to stress james with a small smtp/pop3 client (I used >>> postage, but a simple class with commons-net behind) may be easier and >>> see what happens (eventually with eclipse profiler). >>> >>> It will be for this weekend for me. >>> >>> Bye, >>> >>> Eric >>> >>> >>> On 03/31/2010 07:11 PM, Norman Maurer wrote: >>> >>>> >>>> Hi Eric, >>>> >>>> thx for keeping us in the loop... I'm still not sure why the .m64 >>>> files are still in the tmp folder sometimes.. But I suspect the files >>>> are not the cause of the OOM. >>>> >>>> I didn't get a OOM since yesterday morning (the time I deployed >>>> current trunk version), but just found to new .m64 files.. >>>> >>>> So I'm still searching for the real cause. If nothing helps I will >>>> need to use a profiler to find th leak. >>>> >>>> Bye, >>>> Norman >>>> >>>> 2010/3/31 Eric Charles<[email protected]>: >>>> >>>> >>>>> >>>>> Hi Norman, >>>>> >>>>> I had defined the Null mailet >>>>> >>>>> <mailet match="HostIsLocal" class="Null"> >>>>> <processor> local-address-error</processor> >>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>> </mailet> >>>>> >>>>> but now, I use the standard config >>>>> >>>>> <mailet match="HostIsLocal" class="ToProcessor"> >>>>> <processor> local-address-error</processor> >>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>> </mailet> >>>>> >>>>> I still have the OOM. >>>>> >>>>> I will now deploy a fresh svn with your last commits and enable the >>>>> ValidRcptHandler. >>>>> I keep you posted with the result, >>>>> >>>>> Tks, >>>>> Eric >>>>> >>>>> >>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>>>> >>>>> >>>>>> >>>>>> Hi Eric, >>>>>> >>>>>> you said all the files are related to address-errors , could you show >>>>>> me your address error processor config? >>>>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>>>> the smtpserver.xml file? >>>>>> >>>>>> Thx >>>>>> Norman >>>>>> >>>>>> 2010/3/30, Eric Charles<[email protected]>: >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Oops, no, the files are still there (only [email protected]). >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hi Norman, >>>>>>>> >>>>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>>>> file, on the second after, it is no more there, so the dispose works >>>>>>>> as it should). >>>>>>>> >>>>>>>> I keep you posted with our eventual future OOM. >>>>>>>> >>>>>>>> Tks, >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Norman, >>>>>>>>> >>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so >>>>>>>>> for >>>>>>>>> <[email protected]>). >>>>>>>>> They are from various size (with and without attachment). >>>>>>>>> They are well formed (I can open the downloaded file with >>>>>>>>> thunderbird) >>>>>>>>> >>>>>>>>> However, when I sent a mail to [email protected], I don't see it in >>>>>>>>> the /tmp >>>>>>>>> >>>>>>>>> Tks, >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>>>> could >>>>>>>>>> you check somehow if the .m64 files are files which are related to >>>>>>>>>> successfully delivered mail ? >>>>>>>>>> >>>>>>>>>> Thx, >>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Norman, >>>>>>>>>>> >>>>>>>>>>> Done. Now, we have to wait... >>>>>>>>>>> >>>>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>>>> >>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump >>>>>>>>>>> and >>>>>>>>>>> analyse >>>>>>>>>>> it. >>>>>>>>>>> At the beginning, everything seems ok, and often, when I come >>>>>>>>>>> back, >>>>>>>>>>> the OOM >>>>>>>>>>> has already occured. >>>>>>>>>>> >>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception >>>>>>>>>>> (if >>>>>>>>>>> any, >>>>>>>>>>> let's hope not). >>>>>>>>>>> >>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed >>>>>>>>>>> in >>>>>>>>>>> all >>>>>>>>>>> circumstances. >>>>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>>>> files). >>>>>>>>>>> We should also ensure after that there are no other causes to the >>>>>>>>>>> leaks. >>>>>>>>>>> >>>>>>>>>>> Tks, >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Eric, >>>>>>>>>>>> >>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully >>>>>>>>>>>> this >>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>>>> >>>>>>>>>>>> Thx, >>>>>>>>>>>> Norman >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2010/3/29 Norman Maurer<[email protected]>: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>> >>>>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>>>> (The >>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>>>> thats >>>>>>>>>>>>> a >>>>>>>>>>>>> good pointer. I will try to debug the problem later or >>>>>>>>>>>>> tomorrow. >>>>>>>>>>>>> But I >>>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. >>>>>>>>>>>>> Did >>>>>>>>>>>>> you >>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thx, >>>>>>>>>>>>> Norman >>>>>>>>>>>>> >>>>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>>>> >>>>>>>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>>>> OutOfMemory >>>>>>>>>>>>>> Exception. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and >>>>>>>>>>>>>> always >>>>>>>>>>>>>> found >>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but >>>>>>>>>>>>>> nothing >>>>>>>>>>>>>> helped. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by >>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, >>>>>>>>>>>>>> sometimes >>>>>>>>>>>>>> by a >>>>>>>>>>>>>> activemq class,... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
