Hi,
here new exceptions found in the log. I installed Eclipe Memory Analyzer
but I was not able to open the dump file because it was corrupt. I'll
retry with next OOM.
DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
[email protected] to [email protected]
INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
[email protected] to [email protected]
DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
state root reached
ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
dequeue next message;
nested exception is:
javax.jms.JMSException: PermGen space
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.JMSException: PermGen space
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jms.connection.CachingConnectionFactory
$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
at $Proxy61.createConsumer(Unknown Source)
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
... 2 more
Caused by: java.lang.OutOfMemoryError: PermGen space
at java.lang.String.intern(Native Method)
at
javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
at javax.management.ObjectName.construct(ObjectName.java:644)
at javax.management.ObjectName.<init>(ObjectName.java:1403)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
at
org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
at
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
at org.apache.activemq.broker.TransportConnection
$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at org.apache.activemq.thread.PooledTaskRunner
$1.run(PooledTaskRunner.java:43)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
[email protected] to [email protected]
DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
state root reached
DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
[email protected] to [email protected]
DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
state root reached
ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
INFO 12:27:26,135 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
INFO 12:27:38,052 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
INFO 12:27:38,724 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
INFO 12:28:42,830 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
dequeue next message;
nested exception is:
javax.jms.JMSException: PermGen space
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.JMSException: PermGen space
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jms.connection.CachingConnectionFactory
$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
at $Proxy61.createConsumer(Unknown Source)
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
... 2 more
Caused by: java.lang.OutOfMemoryError: PermGen space
On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
If you go via the menu and open the dump file, it will propose you
(after a few minutes anlaysis) a report that will present you
"suspects". You can copy/paste this report on ml.
There's also YourKit (http://www.yourkit.com/) but I never used it (not
sure it allows to analyze taken dumps, it more for live profiling).
Tks,
- Eric
On 26/05/2011 11:46, Nicola Salvemini wrote:
Hi Eric,
yes, I added "-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
way to analize dump file. Have you any suggestion?
Regards,
Nicola.
On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
Hi,
Getting OOM can have various side effects, so I bet the
java.lang.VerifyError is a result of the OOM.
Did you add "-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/your_path" and got a dump?
If you have now time to analyze to find the "suspect", we can do it for
you if you transfer us the dump (be aware that some objects can contain
some user credentials, but that really doesn't interest us).
Tks,
- Eric
On 26/05/2011 09:20, Nicola Salvemini wrote:
Hello,
the problem persists, but now I have a new exception in the log:
ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
spooling PermGen space
java.lang.OutOfMemoryError: PermGen space
ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
spooling (class: javax/mail/MessagingException, method: superToString
signature: ()Ljava/lang/String;) Illegal constant pool index
java.lang.VerifyError: (class: javax/mail/MessagingException, method:
superToString signature: ()Ljava/lang/String;) Illegal constant pool
index
at
org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
at
org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
at java.lang.Thread.run(Thread.java:662)
ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
spooling PermGen space
java.lang.OutOfMemoryError: PermGen space
Nicola.
On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
If you still encounter the OOM, you can further analyse with "eclipse
mat" [2] for example and ask for "suspects". It will show you the
objects that eat memory (launch james with standard Mx 512M, otherwise
you will need more time to laod dump).
Optionally, you can try to re-add the prefetchPolicy in
james-server-context.xml (see changes from [1]).
Tks,
- Eric
[1] http://s.apache.org/30x
[2] http://www.eclipse.org/mat/
On 25/05/2011 15:56, Nicola Salvemini wrote:
Hi Eric,
I'm using snapshot
james-server-container-spring-3.0-M3-20110523.043027-400
now I'll try to install snapshot
james-server-container-spring-3.0-M3-20110525.115759-407
and I'll add the complete debugging path in the startup script to
further analyse the dump.
Many thanks for your help,
Nicola.
On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
Tks for your answer.
So you've got more or less the same configuration as I have:
- maildir
- low smtp traffic (I've got a bit more with the spam, but it's rejected
via smtp hook)
- not many users
My config here is really stable after weeks (no oom, low memory
consumption when i monitor via JMX, blob-transfer folders empty).
Yes, it sounds like some mails remains blocked, giving the issue.
Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
btw, which version are you using ?
The complete debugging path would be to add
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
startup script and to further analyse the dump: it will give for sure
the cause of oom.
- Eric
On 25/05/2011 15:02, Nicola Salvemini wrote:
On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
Hi,
Hi,
Can you give us a rough estimate of the traffic you have?
Mainly SMTP? How many incoming mails/requests per second?
at this time we don't have very high mail traffic, more or less 500/600
mail per day.
James is configured as smtp and imap servers and I think that most of
the traffic is IMAP (currently there are about ten active clients at the
same time making requests to the James IMAP server).
This will allow us to better define the place to look at.
Also, is the CPU always remain 100% when you have that issue?
Yes, it is in most cases.
Is the log still showing messages after the OOM?
Yes, please consider following log sequence:
DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
noop
DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
I noticed that sometimes the problem occurs for a "blockage" of
ActiveMQ. For some reason it happens that not all spooled mail will be
processed and they remain in "store/activemq/blob-transfer/" folder. If
the number of mail rises too high then the problem occurs.
Last thing: James is installed on a server machine whit following
characteristics:
OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
java version: 1.6.0_25
RAM: 8Gb
Architecture: x86_64
CPU(s): 6
On-line CPU(s) list: 0-5
Thread(s) per core: 1
Core(s) per socket: 6
CPU socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 16
Model: 8
Stepping: 1
CPU MHz: 2599.966
BogoMIPS: 5200.12
Virtualization: AMD-V
Thanks for your help,
Nicola.
Tks,
- Eric
On 25/05/2011 11:54, Nicola Salvemini wrote:
Hello,
I'm using James 3.0-M3 with maildir. Everything works fine for one or
two days until I start having memory problems and I need to restart
James.
The problems begin with a long series of messages such as
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
and then
DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
The CPU usage is over 100% and my only alternative is to restart James.
I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
but the problem keeps coming back.
Does anyone have a suggestion to solve the problem?
Thanks in advance,
Nicola.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
