Re: OutOfMemory exception

Mon, 29 Mar 2010 11:40:44 -0700 

Norman,

Done. Now, we have to wait...

I saw some created *.m64 that were removed.
But there are other ones that remains in /tmp.
I sometimes run a jmap (java memory map) to produce a heap dump and analyse it. At the beginning, everything seems ok, and often, when I come back, the OOM has already occured.
I changed the -Xmx512m to -Xmx256m to have a quicker exception (if any, let's hope not).
The *.m64 are a clue and we should ensure that they are removed in all circumstances. 
This may be the easy part, as we have some visible clues (the files).
We should also ensure after that there are no other causes to the leaks.

Tks,
Eric


On 03/29/2010 08:00 PM, Norman Maurer wrote:

Hi Eric,

I found the cause for the not deleted temporary files. Hopefully this
is the cause of the OOM. Could try to svn up and run again ?

Thx,
Norman


2010/3/29 Norman Maurer<norman.mau...@googlemail.com>:

Hi Eric,

thx for the report. I see exact the same problem today here.. (The
OOM). Didn't notice the files in the tmp folder, but I think thats a
good pointer. I will try to debug the problem later or tomorrow. But I
suspect you are right about the tmp files and jms producers.. Did you
run a kill -3 pid to see what threads are active etc ?

Thx,
Norman

Ps: Patches are welcome :)

2010/3/29 Eric Charles<eric.char...@u-mangate.com>:

Hi,

The last two weeks, I deployed various trunk snapshots.
After james running less than 1 day, I always ran into a OutOfMemory
Exception.

I analysed the logs (for example STACK TRACE 1 in annex), and always found
Camel complaining when trying to deliver on JMS queue.
I tried to adapt the ActiveMQ configuration based on
http://activemq.apache.org/javalangoutofmemory.html but nothing helped.

I also analyzed various heap dump :sometimes, 65% was used by
org.apache.mina.transport.socket.nio.NioSocketSession, sometimes by a
activemq class,...

Yesterday, I had a more explicit stacktrace (see STACK TRACE 2 in annex)
talking about "Too many open files" in the MimeMessageInputStreamSource
constructor when trying file = File.createTempFile(key, ".m64").
Normally, the reference to the file should disappear when invoking the
dispose method (doing a file.delete();).
But, I've got many old files in my /tmp (one more file each minute), so I
suppose that for a reason or another, those files are not deleted.
This could cause the Too many open files problem, leaving the JMS Producers
in memory with any Consumers... ?

Finally, I don't know if the OOM comes from the "Too many open files" or
still from somewhere else ?

Please note that I define the address-error processors as Null (don't want
to keep those mails on disk) and that I have activated the SpamAssassin
mailet.
But even without those extra configs, I've got the OOM.

Any idea,

Eric


STACK TRACE 1

INFO  16:23:01,385 | james.smtpserver | Successfully spooled mail
Mail1269872581383-7178 from johnsmith...@u-mangate.org on 120.56.194.53 for
[johnsmith...@u-mangate.org]
INFO  16:23:01,386 | james.smtpserver | Successfully spooled mail
Mail1269872581383-7179 from catchthism...@u-mangate.net on 120.56.194.53 for
[catchthism...@u-mangate.net]
INFO  16:23:20,887 | james.smtpserver | Successfully spooled mail
Mail1269872600878-7183 from 3edd9016.4070...@u-mangate.be on 120.56.194.53
for [3edd9016.4070...@u-mangate.be]
INFO  16:23:22,007 | james.smtpserver | Successfully spooled mail
Mail1269872602002-7185 from 3c3cd8ce0b.4020...@u-mangate.org on
120.56.194.53 for [3c3cd8ce0b.4020...@u-mangate.org]
INFO  16:23:23,336 | james.smtpserver | Successfully spooled mail
Mail1269872603328-7186 from elizabethsorensen...@bbr-vt.de on 95.169.149.179
for [angelo.bid...@u-mangate.net]
INFO  16:23:34,899 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 1115
INFO  16:23:37,254 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 1240 ms, Index update took 0 ms
INFO  16:23:37,255 | james.smtpserver | Successfully spooled mail
Mail1269872616013-7188 from 3ee83341.3030...@u-mangate.be on 120.56.194.53
for [3ee83341.3030...@u-mangate.be]
INFO  16:23:42,140 | james.smtpserver | Successfully spooled mail
Mail1269872622133-7190 from m...@u-mangate.net on 120.56.194.53 for
[m...@u-mangate.net]
INFO  16:23:56,193 | james.smtpserver | Successfully spooled mail
Mail1269872633794-7195 from shauna.escoba...@superiorsleep.co.nz on
121.147.241.76 for [angelo.bid...@u-mangate.net]
INFO  16:23:56,195 | james.smtpserver | Successfully spooled mail
Mail1269872633798-7196 from 3ee822b7.10...@u-mangate.be on 120.56.194.53 for
[3ee822b7.10...@u-mangate.be]
INFO  16:23:59,740 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 3543 ms, Index update took 0 ms
INFO  16:23:59,743 | james.smtpserver | Successfully spooled mail
Mail1269872636196-7198 from ric.char...@u-mangate.org on 120.56.194.53 for
[ric.char...@u-mangate.org]
INFO  16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 1322 ms, Index update took 0 ms
INFO  16:24:01,066 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 1321 ms, Index update took 0 ms
INFO  16:24:01,069 | james.smtpserver | Successfully spooled mail
Mail1269872639742-7199 from stephane.liu...@u-mangate.net on 120.56.194.53
for [stephane.liu...@u-mangate.net]
INFO  16:24:10,625 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 7100
INFO  16:24:10,626 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 2458 ms, Index update took 7099 ms
INFO  16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 5770
INFO  16:24:19,753 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 3358 ms, Index update took 5769 ms
INFO  16:24:28,832 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 4584
INFO  16:24:53,501 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 22422
INFO  16:24:53,502 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: Journal append took: 0 ms, Index update took 6734 ms
INFO  16:26:27,893 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 87665
ERROR 17:05:55,409 | org.apache.activemq.ActiveMQConnection | Caught an
exception trying to create a JMSException for java.lang.OutOfMemoryError:
Java heap space
java.lang.OutOfMemoryError: Java heap space
INFO  17:06:00,088 | james.smtpserver | Successfully spooled mail
Mail1269872643526-7200 from 8ce0b.4020...@u-mangate.org on 120.56.194.53 for
[8ce0b.4020...@u-mangate.org]
ERROR 17:13:36,768 |
org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
could not poll endpoint: pollingjms://queue?consu
mer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000 caused by:
Java heap space
java.lang.OutOfMemoryError: Java heap space
WARN  17:14:50,711 | org.apache.activemq.broker.TransportConnection.Service
| Async error occurred: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
WARN  17:23:59,872 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of
JMS message listener invoker failed for destination 'processor.transport' -
trying to recover. Cause: Java heap space
WARN  17:28:17,565 | org.apache.activemq.broker.TransportConnection.Service
| Async error occurred: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
WARN  17:27:10,346 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup of
JMS message listener invoker failed for destination 'processor.error' -
trying to recover. Cause: Java heap space
WARN  17:27:05,679 | org.apache.camel.spring.spi.TransactionErrorHandler |
Transaction rollback (0x677bc0ac) for ExchangeId:
ID:srv001-47642-1269835753889-2:41:2:1:158 due exception:
java.util.concurrent.ExecutionException: java.lang.Out
OfMemoryError: Java heap space
ERROR 17:33:57,872 | james.pop3server | WriteRequest holds not a an valid
Object but class java.lang.String
ERROR 17:36:37,234 | james.pop3server | WriteRequest holds not a an valid
Object but class java.lang.String
ERROR 17:37:03,082 | james.pop3server | WriteRequest holds not a an valid
Object but class java.lang.String
ERROR 17:36:37,234 | org.apache.camel.component.jms.EndpointMessageListener
| Caused by: [org.apache.camel.RuntimeCamelException -
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space]
org.apache.camel.RuntimeCamelException:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
    at
org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
    at
org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
    at
org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
    at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.doInTransactionTemplate(TransactionErrorHandler.java:130)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.processInTransaction(TransactionErrorHandler.java:106)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:84)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
    at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:66)
    at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
    at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
    at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:84)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:559)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
    at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323)
    at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:969)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:871)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:199)
    at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:135)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.access$001(TransactionErrorHandler.java:39)
    at
org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:140)
    ... 27 more
Caused by: java.lang.OutOfMemoryError: Java heap space
WARN  17:37:03,084 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Execution
of JMS message listener failed, and no ErrorHandler has been set.
org.apache.camel.RuntimeCamelException:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
    at
org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
    at
org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
    at
org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
    at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.doInTransactionTemplate(TransactionErrorHandler.java:130)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.processInTransaction(TransactionErrorHandler.java:106)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.process(TransactionErrorHandler.java:84)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
    at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:66)
    at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
    at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
    at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:84)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:559)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)
    at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
    at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:323)
    at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:261)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:977)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:969)
    at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:871)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:199)
    at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:135)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
    at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
    at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
    at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)
    at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)
    at
org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)
    at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)
    at
org.apache.camel.spring.spi.TransactionErrorHandler.access$001(TransactionErrorHandler.java:39)
    at
org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:140)
    ... 27 more
Caused by: java.lang.OutOfMemoryError: Java heap space


STACK TRACE 2

WARN  20:58:02,314 | james.smtpserver | Error creating mimemessagesource for
incoming data
javax.mail.MessagingException: Unable to get canonical file path: Too many
open files;
  nested exception is:
    java.io.IOException: Too many open files
    at
org.apache.james.core.MimeMessageInputStreamSource.<init>(MimeMessageInputStreamSource.java:119)
    at
org.apache.james.smtpserver.JamesDataCmdHandler.doDATA(JamesDataCmdHandler.java:74)
    at
org.apache.james.protocols.smtp.core.DataCmdHandler.onCommand(DataCmdHandler.java:93)
    at
org.apache.james.protocols.smtp.core.DataCmdHandler.onCommand(DataCmdHandler.java:43)
    at
org.apache.james.protocols.api.AbstractCommandDispatcher.onLine(AbstractCommandDispatcher.java:140)
    at
org.apache.james.socket.mina.AbstractIoHandler.messageReceived(AbstractIoHandler.java:64)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:713)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
    at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
    at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
    at
org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:375)
    at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:229)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
    at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
    at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:119)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
    at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
    at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:638)
    at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:598)
    at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:587)
    at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:61)
    at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:969)
    at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Too many open files
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.checkAndCreate(File.java:1704)
    at java.io.File.createTempFile(File.java:1792)
    at java.io.File.createTempFile(File.java:1828)
    at
org.apache.james.core.MimeMessageInputStreamSource.<init>(MimeMessageInputStreamSource.java:116)
    ... 38 more
INFO  21:05:16,565 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 1157
INFO  21:07:47,923 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 1337
INFO  21:09:47,703 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 1149
INFO  21:16:19,030 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2476
INFO  21:16:49,026 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2473
INFO  21:17:18,981 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2425
INFO  21:17:49,041 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2489
INFO  21:21:49,245 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 1353
INFO  21:29:21,636 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2549
INFO  21:30:51,385 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2312
INFO  21:31:48,892 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2312
INFO  21:32:24,318 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2495
INFO  21:33:53,627 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 4611
ERROR 21:33:58,247 |
org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aou
tgoing.retry&delay=30000] could not poll endpoint:
pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
caused by: Java heap spac
e
java.lang.OutOfMemoryError: Java heap space
INFO  22:28:40,593 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2301
INFO  22:38:06,343 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2300
INFO  22:44:32,058 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2297
INFO  22:50:28,014 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2302
INFO  23:07:55,152 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2299
INFO  23:18:36,612 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2303
INFO  23:31:58,343 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2299
INFO  23:48:10,277 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2300
INFO  23:53:10,860 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2299
INFO  00:04:12,146 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2298
INFO  00:12:03,122 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2301
INFO  00:48:47,676 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2295
INFO  01:03:29,326 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2301
INFO  01:05:59,689 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2297
INFO  01:21:51,415 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2297
INFO  01:27:32,122 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2322





---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Reply via email to