After reading the page about prefetch limits again, you should maybe set the prefetchlimit to 1 too..
http://activemq.apache.org/what-is-the-prefetch-limit-for.html Pooled Connections and prefetch Consuming messages from a connection pool can be problematic due to prefetch. Unconsumed prefetched messages are only released when a connection is closed, but with a pooled connection the connection close is deferred (for reuse) till the connection pool closes. This leaves prefetched messages unconsumed till the connection is reused. This feature can present as missing or out-of-sequence messages when there is more than one connection in the pool. One solution is to use pooled connections for producers and a non-pooled connection for consumers. This might have performance impacts on the consumer side, if multiple threads try to consume messages at a fast rate. Alternatively, reduce the pool size to 1 for consumers. A third alternative is to reduce the prefetchSize to 1 or 0 with the pooled connection factory. When using Spring JMS and MessageDrivenPojo, you cannot use a prefetch of 0, so use 1 instead. As we use a connection pool... Bye, Norman 2010/4/1 Norman Maurer <[email protected]>: > Hi Eric, > > could you try if this helps... > > In spring-beans.xml replace the jmsConnectionFactory block with: > > <!-- jms connection pooling --> > <bean id="jmsConnectionFactory" > class="org.apache.activemq.pool.PooledConnec > tionFactory" destroy-method="stop"> > <property name="connectionFactory"> > <bean class="org.apache.activemq.ActiveMQConnectionFactory"> > <property name="brokerURL"> > > <value>vm://localhost?broker.useJmx=false&jms.prefetchPolicy.all > =50</value> > </property> > </bean> > </property> > </bean> > > Bye, > Norman > > 2010/4/1 Eric Charles <[email protected]>: >> 2) Launching james, memory usage is 300MB. This seems normal and holding the >> traffic well. At a certain time, memory jumps to 3GB (I gave much memory via >> -Xmx). >> Maybe a certain condition produces this. >> Debugging in eclipse shows me that many threads are created by camel. I also >> had the impression that some threads were falling and other ones were >> recreated. If this is the case, is it normal. >> I'm going to look at activemq/camel (I don't know them really), and also >> still read the http://activemq.apache.org/javalangoutofmemory.html, >> http://activemq.apache.org/my-producer-blocks.html,... >> >> I have the impression that everything is working fine, but at a certain >> moment, there is a condition that makes the memory usage rise extensively >> (not a linear growth, but really a stair). >> >> Bye, >> Eric >> >> >> On 04/01/2010 09:24 AM, Norman Maurer wrote: >>> >>> Hi Eric, >>> >>> thx again for your tests... >>> >>> So I think we have two independent problems here: >>> >>> 1) The .m64 files which are not deleted always: >>> >>> My guess here is that there is something wrong with processing Mail >>> objects with state Null. I will investigate futher.. I also found a >>> possible problem with .m64 files which get not deleted when a >>> Exception is throws while processing the camel routes. I will patch >>> this soon. >>> >>> 2) The OOM >>> >>> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet >>> >>> I think 2) is the more criticial problem and should get higher >>> priority to get fixed.. >>> >>> Bye, >>> Norman >>> >>> >>> >>> 2010/4/1 Eric Charles<[email protected]>: >>> >>>> >>>> Hi Norman, >>>> >>>> Tks for the smpt-source pointer. >>>> I looked at it, but finally, I think I will develop those little classes >>>> to >>>> have more control on it. >>>> I will put it on github. >>>> >>>> I launched latest trunk james yesterday evening. >>>> During the night, 3 hours after the launch, the following occured: >>>> >>>> KahDB slower and slower... >>>> >>>> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2330 >>>> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail. >>>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be >>>> parsed.; >>>> nested exception is: >>>> org.apache.james.imap.mailbox.StorageException: failed. Transaction >>>> commit failed.; >>>> nested exception is: >>>> <openjpa-1.2.1-r752877:753278 fatal store error> >>>> org.apache.openjpa.persistence.RollbackException: The transaction has >>>> been >>>> rolled back. See the nested exceptions for details on the errors that >>>> occurred. >>>> at >>>> >>>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265) >>>> >>>> at >>>> >>>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130) >>>> ... 55 more >>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java >>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt >>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, >>>> contentOctets, >>>> mediaType, subType, textu >>>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) >>>> 1260, >>>> (byte[]) [...@7a00e70c, (long) 47887, (String) text, (String) html, (long) >>>> 876]} [code=0, state=XJ001] >>>> at >>>> >>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192) >>>> at >>>> >>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57) >>>> at >>>> >>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866) >>>> at >>>> >>>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269) >>>> >>>> So still a OOM exception that was shown by yet-another-component (in this >>>> case, the StoreMailbox). >>>> >>>> >>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its >>>> job). >>>> All 4 files were 0 bytes. >>>> >>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB >>>> max >>>> memory). >>>> >>>> With the previous parameters ( -Xmx512m), the process was taking the >>>> whole >>>> 512MB. >>>> >>>> Tks, >>>> Eric >>>> >>>> >>>> On 03/31/2010 08:26 PM, Norman Maurer wrote: >>>> >>>>> >>>>> Hi eric, >>>>> >>>>> thx for all your help, maybe smtp-source ( which is included in >>>>> postfix ) can help you with stress testing. >>>>> >>>>> Bye >>>>> Norman >>>>> >>>>> 2010/3/31, Eric Charles<[email protected]>: >>>>> >>>>> >>>>>> >>>>>> Hi Norman, >>>>>> >>>>>> There are not so much .m64 files, so even it it shows that a mail >>>>>> processing didn't reach the end and may have cause a leak, I don't >>>>>> think >>>>>> it would make crash my process so quick (between 10 and 24 hours). >>>>>> >>>>>> I quickly analysed a few dumps last week : there is always a class that >>>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not >>>>>> (don't remember the details). >>>>>> Also notable, the stack trace (the place where the OOM gives problems) >>>>>> varies. >>>>>> Today stack was: >>>>>> org.apache.camel.RuntimeCamelException: >>>>>> org.apache.camel.CamelExchangeException: Error processing Exchange. >>>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102, >>>>>> responseRequired = true, messageId = >>>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null, >>>>>> originalTransactionId = null, producerId = >>>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination = >>>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp >>>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893, >>>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null, >>>>>> persistent = true, type = null, priority = 4, groupID = null, >>>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID >>>>>> = >>>>>> null, content = org.apache.activemq.util.byteseque...@69b568d0, >>>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter = >>>>>> 0, size = 10718, properties = null, readOnlyProperties = true, >>>>>> readOnlyBody = true, droppable = false}]. Caused by: >>>>>> [java.lang.OutOfMemoryError - Java heap space] >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >>>>>> at >>>>>> >>>>>> >>>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >>>>>> ... >>>>>> >>>>>> >>>>>> >>>>>> I also think the tmp files are not the main cause. >>>>>> >>>>>> I have now james running in eclipse and I made already made a little >>>>>> trip in the code. Quite impressive since last time I looked at it. >>>>>> I will try to stress james with a small smtp/pop3 client (I used >>>>>> postage, but a simple class with commons-net behind) may be easier and >>>>>> see what happens (eventually with eclipse profiler). >>>>>> >>>>>> It will be for this weekend for me. >>>>>> >>>>>> Bye, >>>>>> >>>>>> Eric >>>>>> >>>>>> >>>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote: >>>>>> >>>>>> >>>>>>> >>>>>>> Hi Eric, >>>>>>> >>>>>>> thx for keeping us in the loop... I'm still not sure why the .m64 >>>>>>> files are still in the tmp folder sometimes.. But I suspect the files >>>>>>> are not the cause of the OOM. >>>>>>> >>>>>>> I didn't get a OOM since yesterday morning (the time I deployed >>>>>>> current trunk version), but just found to new .m64 files.. >>>>>>> >>>>>>> So I'm still searching for the real cause. If nothing helps I will >>>>>>> need to use a profiler to find th leak. >>>>>>> >>>>>>> Bye, >>>>>>> Norman >>>>>>> >>>>>>> 2010/3/31 Eric Charles<[email protected]>: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hi Norman, >>>>>>>> >>>>>>>> I had defined the Null mailet >>>>>>>> >>>>>>>> <mailet match="HostIsLocal" class="Null"> >>>>>>>> <processor> local-address-error</processor> >>>>>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>>>>> </mailet> >>>>>>>> >>>>>>>> but now, I use the standard config >>>>>>>> >>>>>>>> <mailet match="HostIsLocal" class="ToProcessor"> >>>>>>>> <processor> local-address-error</processor> >>>>>>>> <notice>550 - Requested action not taken: no such user here</notice> >>>>>>>> </mailet> >>>>>>>> >>>>>>>> I still have the OOM. >>>>>>>> >>>>>>>> I will now deploy a fresh svn with your last commits and enable the >>>>>>>> ValidRcptHandler. >>>>>>>> I keep you posted with the result, >>>>>>>> >>>>>>>> Tks, >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Eric, >>>>>>>>> >>>>>>>>> you said all the files are related to address-errors , could you >>>>>>>>> show >>>>>>>>> me your address error processor config? >>>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in >>>>>>>>> the smtpserver.xml file? >>>>>>>>> >>>>>>>>> Thx >>>>>>>>> Norman >>>>>>>>> >>>>>>>>> 2010/3/30, Eric Charles<[email protected]>: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Oops, no, the files are still there (only [email protected]). >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Norman, >>>>>>>>>>> >>>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess. >>>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1 >>>>>>>>>>> file, on the second after, it is no more there, so the dispose >>>>>>>>>>> works >>>>>>>>>>> as it should). >>>>>>>>>>> >>>>>>>>>>> I keep you posted with our eventual future OOM. >>>>>>>>>>> >>>>>>>>>>> Tks, >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Norman, >>>>>>>>>>>> >>>>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so >>>>>>>>>>>> for >>>>>>>>>>>> <[email protected]>). >>>>>>>>>>>> They are from various size (with and without attachment). >>>>>>>>>>>> They are well formed (I can open the downloaded file with >>>>>>>>>>>> thunderbird) >>>>>>>>>>>> >>>>>>>>>>>> However, when I sent a mail to [email protected], I don't see it >>>>>>>>>>>> in >>>>>>>>>>>> the /tmp >>>>>>>>>>>> >>>>>>>>>>>> Tks, >>>>>>>>>>>> Eric >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>> >>>>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting, >>>>>>>>>>>>> could >>>>>>>>>>>>> you check somehow if the .m64 files are files which are related >>>>>>>>>>>>> to >>>>>>>>>>>>> successfully delivered mail ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thx, >>>>>>>>>>>>> Norman >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Norman, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Done. Now, we have to wait... >>>>>>>>>>>>>> >>>>>>>>>>>>>> I saw some created *.m64 that were removed. >>>>>>>>>>>>>> But there are other ones that remains in /tmp. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump >>>>>>>>>>>>>> and >>>>>>>>>>>>>> analyse >>>>>>>>>>>>>> it. >>>>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come >>>>>>>>>>>>>> back, >>>>>>>>>>>>>> the OOM >>>>>>>>>>>>>> has already occured. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception >>>>>>>>>>>>>> (if >>>>>>>>>>>>>> any, >>>>>>>>>>>>>> let's hope not). >>>>>>>>>>>>>> >>>>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed >>>>>>>>>>>>>> in >>>>>>>>>>>>>> all >>>>>>>>>>>>>> circumstances. >>>>>>>>>>>>>> This may be the easy part, as we have some visible clues (the >>>>>>>>>>>>>> files). >>>>>>>>>>>>>> We should also ensure after that there are no other causes to >>>>>>>>>>>>>> the >>>>>>>>>>>>>> leaks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tks, >>>>>>>>>>>>>> Eric >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I found the cause for the not deleted temporary files. >>>>>>>>>>>>>>> Hopefully >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thx, >>>>>>>>>>>>>>> Norman >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2010/3/29 Norman Maurer<[email protected]>: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thx for the report. I see exact the same problem today here.. >>>>>>>>>>>>>>>> (The >>>>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think >>>>>>>>>>>>>>>> thats >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> good pointer. I will try to debug the problem later or >>>>>>>>>>>>>>>> tomorrow. >>>>>>>>>>>>>>>> But I >>>>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. >>>>>>>>>>>>>>>> Did >>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thx, >>>>>>>>>>>>>>>> Norman >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ps: Patches are welcome :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2010/3/29 Eric Charles<[email protected]>: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots. >>>>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a >>>>>>>>>>>>>>>>> OutOfMemory >>>>>>>>>>>>>>>>> Exception. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>> found >>>>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue. >>>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on >>>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but >>>>>>>>>>>>>>>>> nothing >>>>>>>>>>>>>>>>> helped. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used >>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, >>>>>>>>>>>>>>>>> sometimes >>>>>>>>>>>>>>>>> by a >>>>>>>>>>>>>>>>> activemq class,... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
