Hi Eric, sure.. we have to find the OOM cause. What would be interesting, could you check somehow if the .m64 files are files which are related to successfully delivered mail ?
Thx, Norman 2010/3/29 Eric Charles <eric.char...@u-mangate.com>: > Norman, > > Done. Now, we have to wait... > > I saw some created *.m64 that were removed. > But there are other ones that remains in /tmp. > > I sometimes run a jmap (java memory map) to produce a heap dump and analyse > it. > At the beginning, everything seems ok, and often, when I come back, the OOM > has already occured. > > I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, > let's hope not). > > The *.m64 are a clue and we should ensure that they are removed in all > circumstances. > This may be the easy part, as we have some visible clues (the files). > We should also ensure after that there are no other causes to the leaks. > > Tks, > Eric > > > On 03/29/2010 08:00 PM, Norman Maurer wrote: >> >> Hi Eric, >> >> I found the cause for the not deleted temporary files. Hopefully this >> is the cause of the OOM. Could try to svn up and run again ? >> >> Thx, >> Norman >> >> >> 2010/3/29 Norman Maurer<norman.mau...@googlemail.com>: >> >>> >>> Hi Eric, >>> >>> thx for the report. I see exact the same problem today here.. (The >>> OOM). Didn't notice the files in the tmp folder, but I think thats a >>> good pointer. I will try to debug the problem later or tomorrow. But I >>> suspect you are right about the tmp files and jms producers.. Did you >>> run a kill -3 pid to see what threads are active etc ? >>> >>> Thx, >>> Norman >>> >>> Ps: Patches are welcome :) >>> >>> 2010/3/29 Eric Charles<eric.char...@u-mangate.com>: >>> >>>> >>>> Hi, >>>> >>>> The last two weeks, I deployed various trunk snapshots. >>>> After james running less than 1 day, I always ran into a OutOfMemory >>>> Exception. >>>> >>>> I analysed the logs (for example STACK TRACE 1 in annex), and always >>>> found >>>> Camel complaining when trying to deliver on JMS queue. >>>> I tried to adapt the ActiveMQ configuration based on >>>> http://activemq.apache.org/javalangoutofmemory.html but nothing helped. >>>> >>>> I also analyzed various heap dump :sometimes, 65% was used by >>>> org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a >>>> activemq class,... >>>> >>>> Yesterday, I had a more explicit stacktrace (see STACK TRACE 2 in annex) >>>> talking about "Too many open files" in the MimeMessageInputStreamSource >>>> constructor when trying file = File.createTempFile(key, ".m64"). >>>> Normally, the reference to the file should disappear when invoking the >>>> dispose method (doing a file.delete();). >>>> But, I've got many old files in my /tmp (one more file each minute), so >>>> I >>>> suppose that for a reason or another, those files are not deleted. >>>> This could cause the Too many open files problem, leaving the JMS >>>> Producers >>>> in memory with any Consumers... ? >>>> >>>> Finally, I don't know if the OOM comes from the "Too many open files" or >>>> still from somewhere else ? >>>> >>>> Please note that I define the address-error processors as Null (don't >>>> want >>>> to keep those mails on disk) and that I have activated the SpamAssassin >>>> mailet. >>>> But even without those extra configs, I've got the OOM. >>>> >>>> Any idea, >>>> >>>> Eric >>>> >>>> >>>> STACK TRACE 1 >>>> >>>> INFO 16:23:01,385 | james.smtpserver | Successfully spooled mail >>>> Mail1269872581383-7178 from johnsmith...@u-mangate.org on 120.56.194.53 >>>> for >>>> [johnsmith...@u-mangate.org] >>>> INFO 16:23:01,386 | james.smtpserver | Successfully spooled mail >>>> Mail1269872581383-7179 from catchthism...@u-mangate.net on 120.56.194.53 >>>> for >>>> [catchthism...@u-mangate.net] >>>> INFO 16:23:20,887 | james.smtpserver | Successfully spooled mail >>>> Mail1269872600878-7183 from 3edd9016.4070...@u-mangate.be on >>>> 120.56.194.53 >>>> for [3edd9016.4070...@u-mangate.be] >>>> INFO 16:23:22,007 | james.smtpserver | Successfully spooled mail >>>> Mail1269872602002-7185 from 3c3cd8ce0b.4020...@u-mangate.org on >>>> 120.56.194.53 for [3c3cd8ce0b.4020...@u-mangate.org] >>>> INFO 16:23:23,336 | james.smtpserver | Successfully spooled mail >>>> Mail1269872603328-7186 from elizabethsorensen...@bbr-vt.de on >>>> 95.169.149.179 >>>> for [angelo.bid...@u-mangate.net] >>>> INFO 16:23:34,899 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 1115 >>>> INFO 16:23:37,254 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 1240 ms, Index update took 0 ms >>>> INFO 16:23:37,255 | james.smtpserver | Successfully spooled mail >>>> Mail1269872616013-7188 from 3ee83341.3030...@u-mangate.be on >>>> 120.56.194.53 >>>> for [3ee83341.3030...@u-mangate.be] >>>> INFO 16:23:42,140 | james.smtpserver | Successfully spooled mail >>>> Mail1269872622133-7190 from m...@u-mangate.net on 120.56.194.53 for >>>> [m...@u-mangate.net] >>>> INFO 16:23:56,193 | james.smtpserver | Successfully spooled mail >>>> Mail1269872633794-7195 from shauna.escoba...@superiorsleep.co.nz on >>>> 121.147.241.76 for [angelo.bid...@u-mangate.net] >>>> INFO 16:23:56,195 | james.smtpserver | Successfully spooled mail >>>> Mail1269872633798-7196 from 3ee822b7.10...@u-mangate.be on 120.56.194.53 >>>> for >>>> [3ee822b7.10...@u-mangate.be] >>>> INFO 16:23:59,740 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 3543 ms, Index update took 0 ms >>>> INFO 16:23:59,743 | james.smtpserver | Successfully spooled mail >>>> Mail1269872636196-7198 from ric.char...@u-mangate.org on 120.56.194.53 >>>> for >>>> [ric.char...@u-mangate.org] >>>> INFO 16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 1322 ms, Index update took 0 ms >>>> INFO 16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 1321 ms, Index update took 0 ms >>>> INFO 16:24:01,069 | james.smtpserver | Successfully spooled mail >>>> Mail1269872639742-7199 from stephane.liu...@u-mangate.net on >>>> 120.56.194.53 >>>> for [stephane.liu...@u-mangate.net] >>>> INFO 16:24:10,625 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 7100 >>>> INFO 16:24:10,626 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 2458 ms, Index update took 7099 ms >>>> INFO 16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 5770 >>>> INFO 16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 3358 ms, Index update took 5769 ms >>>> INFO 16:24:28,832 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 4584 >>>> INFO 16:24:53,501 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 22422 >>>> INFO 16:24:53,502 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: Journal append took: 0 ms, Index update took 6734 ms >>>> INFO 16:26:27,893 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 87665 >>>> ERROR 17:05:55,409 | org.apache.activemq.ActiveMQConnection | Caught an >>>> exception trying to create a JMSException for >>>> java.lang.OutOfMemoryError: >>>> Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> INFO 17:06:00,088 | james.smtpserver | Successfully spooled mail >>>> Mail1269872643526-7200 from 8ce0b.4020...@u-mangate.org on 120.56.194.53 >>>> for >>>> [8ce0b.4020...@u-mangate.org] >>>> ERROR 17:13:36,768 | >>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer >>>> >>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000] >>>> could not poll endpoint: pollingjms://queue?consu >>>> mer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000 caused by: >>>> Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> WARN 17:14:50,711 | >>>> org.apache.activemq.broker.TransportConnection.Service >>>> | Async error occurred: java.lang.OutOfMemoryError: Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> WARN 17:23:59,872 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of >>>> JMS message listener invoker failed for destination >>>> 'processor.transport' - >>>> trying to recover. Cause: Java heap space >>>> WARN 17:28:17,565 | >>>> org.apache.activemq.broker.TransportConnection.Service >>>> | Async error occurred: java.lang.OutOfMemoryError: Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> WARN 17:27:10,346 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of >>>> JMS message listener invoker failed for destination 'processor.error' - >>>> trying to recover. Cause: Java heap space >>>> WARN 17:27:05,679 | org.apache.camel.spring.spi.TransactionErrorHandler >>>> | >>>> Transaction rollback (0x677bc0ac) for ExchangeId: >>>> ID:srv001-47642-1269835753889-2:41:2:1:158 due exception: >>>> java.util.concurrent.ExecutionException: java.lang.Out >>>> OfMemoryError: Java heap space >>>> ERROR 17:33:57,872 | james.pop3server | WriteRequest holds not a an >>>> valid >>>> Object but class java.lang.String >>>> ERROR 17:36:37,234 | james.pop3server | WriteRequest holds not a an >>>> valid >>>> Object but class java.lang.String >>>> ERROR 17:37:03,082 | james.pop3server | WriteRequest holds not a an >>>> valid >>>> Object but class java.lang.String >>>> ERROR 17:36:37,234 | >>>> org.apache.camel.component.jms.EndpointMessageListener >>>> | Caused by: [org.apache.camel.RuntimeCamelException - >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>> Java >>>> heap space] >>>> org.apache.camel.RuntimeCamelException: >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>> Java >>>> heap space >>>> at >>>> >>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >>>> at >>>> >>>> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33) >>>> at >>>> >>>> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.doInTransactionTemplate(TransactionErrorHandler.java:130) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.processInTransaction(TransactionErrorHandler.java:106) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:84) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>> at >>>> >>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:66) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>> at >>>> >>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>> at >>>> >>>> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:84) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:559) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:969) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:871) >>>> at java.lang.Thread.run(Thread.java:619) >>>> Caused by: java.util.concurrent.ExecutionException: >>>> java.lang.OutOfMemoryError: Java heap space >>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >>>> at >>>> >>>> org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:199) >>>> at >>>> >>>> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:135) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.access$001(TransactionErrorHandler.java:39) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:140) >>>> ... 27 more >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> WARN 17:37:03,084 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | >>>> Execution >>>> of JMS message listener failed, and no ErrorHandler has been set. >>>> org.apache.camel.RuntimeCamelException: >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>> Java >>>> heap space >>>> at >>>> >>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154) >>>> at >>>> >>>> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33) >>>> at >>>> >>>> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.doInTransactionTemplate(TransactionErrorHandler.java:130) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.processInTransaction(TransactionErrorHandler.java:106) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:84) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>> at >>>> >>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:66) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48) >>>> at >>>> >>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67) >>>> at >>>> >>>> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:84) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:559) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323) >>>> at >>>> >>>> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:969) >>>> at >>>> >>>> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:871) >>>> at java.lang.Thread.run(Thread.java:619) >>>> Caused by: java.util.concurrent.ExecutionException: >>>> java.lang.OutOfMemoryError: Java heap space >>>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >>>> at >>>> >>>> org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:199) >>>> at >>>> >>>> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:135) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53) >>>> at >>>> >>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82) >>>> at >>>> >>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49) >>>> at >>>> >>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143) >>>> at >>>> >>>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler.access$001(TransactionErrorHandler.java:39) >>>> at >>>> >>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:140) >>>> ... 27 more >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> >>>> >>>> STACK TRACE 2 >>>> >>>> WARN 20:58:02,314 | james.smtpserver | Error creating mimemessagesource >>>> for >>>> incoming data >>>> javax.mail.MessagingException: Unable to get canonical file path: Too >>>> many >>>> open files; >>>> nested exception is: >>>> java.io.IOException: Too many open files >>>> at >>>> >>>> org.apache.james.core.MimeMessageInputStreamSource.<init>(MimeMessageInputStreamSource.java:119) >>>> at >>>> >>>> org.apache.james.smtpserver.JamesDataCmdHandler.doDATA(JamesDataCmdHandler.java:74) >>>> at >>>> >>>> org.apache.james.protocols.smtp.core.DataCmdHandler.onCommand(DataCmdHandler.java:93) >>>> at >>>> >>>> org.apache.james.protocols.smtp.core.DataCmdHandler.onCommand(DataCmdHandler.java:43) >>>> at >>>> >>>> org.apache.james.protocols.api.AbstractCommandDispatcher.onLine(AbstractCommandDispatcher.java:140) >>>> at >>>> >>>> org.apache.james.socket.mina.AbstractIoHandler.messageReceived(AbstractIoHandler.java:64) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:713) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793) >>>> at >>>> >>>> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793) >>>> at >>>> >>>> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793) >>>> at >>>> >>>> org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:375) >>>> at >>>> >>>> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:229) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793) >>>> at >>>> >>>> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793) >>>> at >>>> >>>> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434) >>>> at >>>> >>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426) >>>> at >>>> >>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:638) >>>> at >>>> >>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:598) >>>> at >>>> >>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:587) >>>> at >>>> >>>> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:61) >>>> at >>>> >>>> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:969) >>>> at >>>> >>>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:619) >>>> Caused by: java.io.IOException: Too many open files >>>> at java.io.UnixFileSystem.createFileExclusively(Native Method) >>>> at java.io.File.checkAndCreate(File.java:1704) >>>> at java.io.File.createTempFile(File.java:1792) >>>> at java.io.File.createTempFile(File.java:1828) >>>> at >>>> >>>> org.apache.james.core.MimeMessageInputStreamSource.<init>(MimeMessageInputStreamSource.java:116) >>>> ... 38 more >>>> INFO 21:05:16,565 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 1157 >>>> INFO 21:07:47,923 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 1337 >>>> INFO 21:09:47,703 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 1149 >>>> INFO 21:16:19,030 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2476 >>>> INFO 21:16:49,026 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2473 >>>> INFO 21:17:18,981 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2425 >>>> INFO 21:17:49,041 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2489 >>>> INFO 21:21:49,245 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 1353 >>>> INFO 21:29:21,636 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2549 >>>> INFO 21:30:51,385 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2312 >>>> INFO 21:31:48,892 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2312 >>>> INFO 21:32:24,318 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2495 >>>> INFO 21:33:53,627 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 4611 >>>> ERROR 21:33:58,247 | >>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer >>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aou >>>> tgoing.retry&delay=30000] could not poll endpoint: >>>> >>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000 >>>> caused by: Java heap spac >>>> e >>>> java.lang.OutOfMemoryError: Java heap space >>>> INFO 22:28:40,593 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2301 >>>> INFO 22:38:06,343 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2300 >>>> INFO 22:44:32,058 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2297 >>>> INFO 22:50:28,014 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2302 >>>> INFO 23:07:55,152 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2299 >>>> INFO 23:18:36,612 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2303 >>>> INFO 23:31:58,343 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2299 >>>> INFO 23:48:10,277 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2300 >>>> INFO 23:53:10,860 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2299 >>>> INFO 00:04:12,146 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2298 >>>> INFO 00:12:03,122 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2301 >>>> INFO 00:48:47,676 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2295 >>>> INFO 01:03:29,326 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2301 >>>> INFO 01:05:59,689 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2297 >>>> INFO 01:21:51,415 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2297 >>>> INFO 01:27:32,122 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow >>>> KahaDB access: cleanup took 2322 >>>> >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>>> >>>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
