Hi, here new exceptions found in the log. I installed Eclipe Memory Analyzer but I was not able to open the dump file because it was corrupt. I'll retry with next OOM.
DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92==== DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport INFO 12:21:18,821 | james.mailetcontext | Local delivered mail Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from [email protected] to [email protected] INFO 12:21:20,157 | james.mailetcontext | Local delivered mail Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from [email protected] to [email protected] DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for state root reached ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next message; nested exception is: javax.jms.JMSException: PermGen space at org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171) at org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135) at java.lang.Thread.run(Thread.java:662) Caused by: javax.jms.JMSException: PermGen space at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287) at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874) at org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.jms.connection.CachingConnectionFactory $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) at $Proxy61.createConsumer(Unknown Source) at org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109) ... 2 more Caused by: java.lang.OutOfMemoryError: PermGen space at java.lang.String.intern(Native Method) at javax.management.ObjectName.setCanonicalName(ObjectName.java:816) at javax.management.ObjectName.construct(ObjectName.java:644) at javax.management.ObjectName.<init>(ObjectName.java:1403) at org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184) at org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45) at org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268) at org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444) at org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240) at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) at org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91) at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) at org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95) at org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550) at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311) at org.apache.activemq.broker.TransportConnection $1.onCommand(TransportConnection.java:185) at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219) at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) at org.apache.activemq.thread.PooledTaskRunner $1.run(PooledTaskRunner.java:43) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) ... 1 more DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1==== DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport INFO 12:22:16,207 | james.mailetcontext | Local delivered mail Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from [email protected] to [email protected] DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for state root reached DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc==== DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport INFO 12:22:22,873 | james.mailetcontext | Local delivered mail Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from [email protected] to [email protected] DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for state root reached ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message INFO 12:27:26,135 | james.mailetcontext | Exception caught in RemoteDelivery.run() INFO 12:27:38,052 | james.mailetcontext | Exception caught in RemoteDelivery.run() INFO 12:27:38,724 | james.mailetcontext | Exception caught in RemoteDelivery.run() ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message INFO 12:28:42,830 | james.mailetcontext | Exception caught in RemoteDelivery.run() ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while spooling Unable to dequeue next message org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next message; nested exception is: javax.jms.JMSException: PermGen space at org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171) at org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135) at java.lang.Thread.run(Thread.java:662) Caused by: javax.jms.JMSException: PermGen space at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287) at org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874) at org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060) at org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973) at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.jms.connection.CachingConnectionFactory $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) at $Proxy61.createConsumer(Unknown Source) at org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109) ... 2 more Caused by: java.lang.OutOfMemoryError: PermGen space On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote: > I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/). > If you go via the menu and open the dump file, it will propose you > (after a few minutes anlaysis) a report that will present you > "suspects". You can copy/paste this report on ml. > > There's also YourKit (http://www.yourkit.com/) but I never used it (not > sure it allows to analyze taken dumps, it more for live profiling). > > Tks, > - Eric > > > On 26/05/2011 11:46, Nicola Salvemini wrote: > > Hi Eric, > > > > yes, I added "-XX:+HeapDumpOnOutOfMemoryError > > -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy > > way to analize dump file. Have you any suggestion? > > > > Regards, > > Nicola. > > > > > > On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote: > >> Hi, > >> Getting OOM can have various side effects, so I bet the > >> java.lang.VerifyError is a result of the OOM. > >> > >> Did you add "-XX:+HeapDumpOnOutOfMemoryError > >> -XX:HeapDumpPath=/your_path" and got a dump? > >> > >> If you have now time to analyze to find the "suspect", we can do it for > >> you if you transfer us the dump (be aware that some objects can contain > >> some user credentials, but that really doesn't interest us). > >> > >> Tks, > >> - Eric > >> > >> > >> On 26/05/2011 09:20, Nicola Salvemini wrote: > >>> Hello, > >>> > >>> the problem persists, but now I have a new exception in the log: > >>> > >>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while > >>> spooling PermGen space > >>> java.lang.OutOfMemoryError: PermGen space > >>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while > >>> spooling (class: javax/mail/MessagingException, method: superToString > >>> signature: ()Ljava/lang/String;) Illegal constant pool index > >>> java.lang.VerifyError: (class: javax/mail/MessagingException, method: > >>> superToString signature: ()Ljava/lang/String;) Illegal constant pool > >>> index > >>> at > >>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68) > >>> at > >>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104) > >>> at > >>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147) > >>> at java.lang.Thread.run(Thread.java:662) > >>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while > >>> spooling PermGen space > >>> java.lang.OutOfMemoryError: PermGen space > >>> > >>> > >>> Nicola. > >>> > >>> > >>> > >>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote: > >>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of > >>>> 20/5. > >>>> > >>>> If you still encounter the OOM, you can further analyse with "eclipse > >>>> mat" [2] for example and ask for "suspects". It will show you the > >>>> objects that eat memory (launch james with standard Mx 512M, otherwise > >>>> you will need more time to laod dump). > >>>> > >>>> Optionally, you can try to re-add the prefetchPolicy in > >>>> james-server-context.xml (see changes from [1]). > >>>> > >>>> Tks, > >>>> - Eric > >>>> > >>>> [1] http://s.apache.org/30x > >>>> [2] http://www.eclipse.org/mat/ > >>>> > >>>> On 25/05/2011 15:56, Nicola Salvemini wrote: > >>>>> Hi Eric, > >>>>> I'm using snapshot > >>>>> james-server-container-spring-3.0-M3-20110523.043027-400 > >>>>> > >>>>> now I'll try to install snapshot > >>>>> james-server-container-spring-3.0-M3-20110525.115759-407 > >>>>> and I'll add the complete debugging path in the startup script to > >>>>> further analyse the dump. > >>>>> > >>>>> Many thanks for your help, > >>>>> > >>>>> Nicola. > >>>>> > >>>>> > >>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote: > >>>>>> Tks for your answer. > >>>>>> > >>>>>> So you've got more or less the same configuration as I have: > >>>>>> - maildir > >>>>>> - low smtp traffic (I've got a bit more with the spam, but it's > >>>>>> rejected > >>>>>> via smtp hook) > >>>>>> - not many users > >>>>>> > >>>>>> My config here is really stable after weeks (no oom, low memory > >>>>>> consumption when i monitor via JMX, blob-transfer folders empty). > >>>>>> > >>>>>> Yes, it sounds like some mails remains blocked, giving the issue. > >>>>>> Would it come from recent > >>>>>> https://issues.apache.org/jira/browse/JAMES-1253 ? > >>>>>> btw, which version are you using ? > >>>>>> > >>>>>> The complete debugging path would be to add > >>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the > >>>>>> startup script and to further analyse the dump: it will give for sure > >>>>>> the cause of oom. > >>>>>> > >>>>>> - Eric > >>>>>> > >>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote: > >>>>>>> > >>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote: > >>>>>>>> Hi, > >>>>>>> Hi, > >>>>>>> > >>>>>>>> > >>>>>>>> Can you give us a rough estimate of the traffic you have? > >>>>>>>> Mainly SMTP? How many incoming mails/requests per second? > >>>>>>>> > >>>>>>> > >>>>>>> at this time we don't have very high mail traffic, more or less > >>>>>>> 500/600 > >>>>>>> mail per day. > >>>>>>> James is configured as smtp and imap servers and I think that most of > >>>>>>> the traffic is IMAP (currently there are about ten active clients at > >>>>>>> the > >>>>>>> same time making requests to the James IMAP server). > >>>>>>> > >>>>>>>> This will allow us to better define the place to look at. > >>>>>>>> > >>>>>>>> Also, is the CPU always remain 100% when you have that issue? > >>>>>>> > >>>>>>> Yes, it is in most cases. > >>>>>>> > >>>>>>>> Is the log still showing messages after the OOM? > >>>>>>> > >>>>>>> Yes, please consider following log sequence: > >>>>>>> > >>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request > >>>>>>> java.lang.OutOfMemoryError: PermGen space > >>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125 > >>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>: > >>>>>>> noop > >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook > >>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb > >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request > >>>>>>> java.lang.OutOfMemoryError: PermGen space > >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request > >>>>>>> java.lang.OutOfMemoryError: PermGen space > >>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request > >>>>>>> java.lang.OutOfMemoryError: PermGen space > >>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974 > >>>>>>> > >>>>>>> > >>>>>>> I noticed that sometimes the problem occurs for a "blockage" of > >>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be > >>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. > >>>>>>> If > >>>>>>> the number of mail rises too high then the problem occurs. > >>>>>>> > >>>>>>> Last thing: James is installed on a server machine whit following > >>>>>>> characteristics: > >>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5 > >>>>>>> java version: 1.6.0_25 > >>>>>>> RAM: 8Gb > >>>>>>> Architecture: x86_64 > >>>>>>> CPU(s): 6 > >>>>>>> On-line CPU(s) list: 0-5 > >>>>>>> Thread(s) per core: 1 > >>>>>>> Core(s) per socket: 6 > >>>>>>> CPU socket(s): 1 > >>>>>>> Vendor ID: AuthenticAMD > >>>>>>> CPU family: 16 > >>>>>>> Model: 8 > >>>>>>> Stepping: 1 > >>>>>>> CPU MHz: 2599.966 > >>>>>>> BogoMIPS: 5200.12 > >>>>>>> Virtualization: AMD-V > >>>>>>> > >>>>>>> > >>>>>>> Thanks for your help, > >>>>>>> > >>>>>>> Nicola. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> Tks, > >>>>>>>> - Eric > >>>>>>>> > >>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote: > >>>>>>>>> > >>>>>>>>> Hello, > >>>>>>>>> > >>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one > >>>>>>>>> or > >>>>>>>>> two days until I start having memory problems and I need to restart > >>>>>>>>> James. > >>>>>>>>> The problems begin with a long series of messages such as > >>>>>>>>> > >>>>>>>>> WARN 04:46:27,928 | > >>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker > >>>>>>>>> | Failed to register MBean: > >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 > >>>>>>>>> WARN 04:46:27,928 | > >>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker > >>>>>>>>> | Failed to register MBean: > >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 > >>>>>>>>> WARN 04:46:27,928 | > >>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker > >>>>>>>>> | Failed to register MBean: > >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 > >>>>>>>>> WARN 04:46:27,928 | > >>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker > >>>>>>>>> | Failed to register MBean: > >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 > >>>>>>>>> > >>>>>>>>> and then > >>>>>>>>> > >>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request > >>>>>>>>> java.lang.OutOfMemoryError: PermGen space > >>>>>>>>> > >>>>>>>>> The CPU usage is over 100% and my only alternative is to restart > >>>>>>>>> James. > >>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m > >>>>>>>>> -Xmx1024m) > >>>>>>>>> but the problem keeps coming back. > >>>>>>>>> Does anyone have a suggestion to solve the problem? > >>>>>>>>> Thanks in advance, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Nicola. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>> For additional commands, e-mail: [email protected] > >>>>>>> > >>>>>> > >>>>>> > >>>>>> --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: [email protected] > >>>>> For additional commands, e-mail: [email protected] > >>>>> > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: [email protected] > >>>> For additional commands, e-mail: [email protected] > >>>> > >>> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
