Hi Eric, great! Thanks to you, for your time and your support. Bye, Nicola.
2011/5/29 Eric Charles <[email protected]>: > Hi Nicola, > > I've committed a patch. > It could take one day for the snapshots to be available. > > I thank you very much for your help and the follow on this :) > - Eric > > On 27/05/2011 17:38, Nicola Salvemini wrote: >> >> Hello Eric, >> >> I opened JIRA JAMES-1259. >> >> Thank you very much for everything, best regards, >> >> Nicola. >> >> >> >> On Fri, 2011-05-27 at 16:36 +0200, Eric Charles wrote: >>> >>> Hi Nicola, >>> >>> Yes, you're right. >>> >>> The problem arise in fact when the mail must be delivered to/cc local >>> and non-local recipients, so when LocalDelivery and RemoteDelivery >>> mailets must come into the play for the same mail. >>> >>> I see cpu rising to 100% and leaking files in blob transfer - I didn't >>> wait to see oom, but I'm pretty sure it will come. >>> >>> I also tested with remotedelivery configured with a gateway and without >>> a gateway (in my case, non gateway make the mail undeliverable to the >>> remote recipient). This does not change anything (bug is there for both >>> configuration). >>> >>> Could I ask you to open a JIRA >>> (https://issues.apache.org/jira/browse/JAMES) for this and define it as >>> blocker. >>> >>> Tks very much for the follow, >>> - Eric >>> >>> On 27/05/2011 15:09, Nicola Salvemini wrote: >>>> >>>> Hello, >>>> my mailetcontainer.xml is very standard, I don't use custom servlet or >>>> something strange. >>>> Could you please try sending only with James without using your ISP SMTP >>>> server? >>>> >>>> Thanks, >>>> >>>> Nicola. >>>> >>>> >>>> >>>> >>>> On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote: >>>>> >>>>> Replicated issue is the begin of the solution :) >>>>> >>>>> I sent a mail with my account in cc, and everything happens fine >>>>> (delivered, no remaining files in blob-transfer). >>>>> >>>>> It goes via my ISP SMTP server before reaching James, but that should >>>>> not change anything. >>>>> >>>>> Did you change mailetcontainer.xml or any other configuration ? >>>>> >>>>> Tks, >>>>> - Eric >>>>> >>>>> On 27/05/2011 12:34, Nicola Salvemini wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have a good news! I'm able to replicate problem: I noticed the >>>>>> problem >>>>>> appears when James try to delivery email having a local address in CC. >>>>>> Please consider following example: >>>>>> James is configured as SMTP server for mydomain.com. If it try to >>>>>> deliver email to [email protected] having [email protected] in CC, >>>>>> this >>>>>> is the result: >>>>>> >>>>>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail >>>>>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1==== >>>>>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root >>>>>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor >>>>>> transport >>>>>> >>>>>> Nothing more! >>>>>> >>>>>>> From this point on, James is not able to send any mail. All emails >>>>>> >>>>>> remain in "store/activemq/blob-transfer/" until OOM is reached. >>>>>> To resolve it's necessary delete email in >>>>>> "store/activemq/blob-transfer/" and restart the server. >>>>>> >>>>>> I think the problem is this. What do you think? >>>>>> >>>>>> Nicola. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote: >>>>>>> >>>>>>> OK, let's see what the next dump tells us. >>>>>>> >>>>>>> Once a OOM occurs, the consecutive exceptions can come from anywhere >>>>>>> and >>>>>>> should not be considered too much. Memory becomes short, jvm is >>>>>>> confused... >>>>>>> >>>>>>> Have you still some messages stuck in queue and remaining files in >>>>>>> "store/activemq/blob-transfer/" ? >>>>>>> >>>>>>> I was wondering about your mailetcontainer.xml configuration. Did you >>>>>>> configure something special? Is it possible that for some reasones, >>>>>>> the >>>>>>> processor route a mails takes is "infinite" ? >>>>>>> >>>>>>> Tks, >>>>>>> - Eric >>>>>>> >>>>>>> >>>>>>> On 26/05/2011 15:18, Nicola Salvemini wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> here new exceptions found in the log. I installed Eclipe Memory >>>>>>>> Analyzer >>>>>>>> but I was not able to open the dump file because it was corrupt. >>>>>>>> I'll >>>>>>>> retry with next OOM. >>>>>>>> >>>>>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail >>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92==== >>>>>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root >>>>>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor >>>>>>>> transport >>>>>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail >>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully >>>>>>>> from >>>>>>>> [email protected] to [email protected] >>>>>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail >>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully >>>>>>>> from >>>>>>>> [email protected] to [email protected] >>>>>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor >>>>>>>> for >>>>>>>> state root reached >>>>>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to >>>>>>>> dequeue next message; >>>>>>>> nested exception is: >>>>>>>> javax.jms.JMSException: PermGen space >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135) >>>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>>> Caused by: javax.jms.JMSException: PermGen space >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973) >>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown >>>>>>>> Source) >>>>>>>> at >>>>>>>> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>>>> at >>>>>>>> org.springframework.jms.connection.CachingConnectionFactory >>>>>>>> >>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) >>>>>>>> at $Proxy61.createConsumer(Unknown Source) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109) >>>>>>>> ... 2 more >>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space >>>>>>>> at java.lang.String.intern(Native Method) >>>>>>>> at >>>>>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816) >>>>>>>> at >>>>>>>> javax.management.ObjectName.construct(ObjectName.java:644) >>>>>>>> at >>>>>>>> javax.management.ObjectName.<init>(ObjectName.java:1403) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311) >>>>>>>> at org.apache.activemq.broker.TransportConnection >>>>>>>> $1.onCommand(TransportConnection.java:185) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) >>>>>>>> at org.apache.activemq.thread.PooledTaskRunner >>>>>>>> $1.run(PooledTaskRunner.java:43) >>>>>>>> at java.util.concurrent.ThreadPoolExecutor >>>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886) >>>>>>>> at java.util.concurrent.ThreadPoolExecutor >>>>>>>> $Worker.run(ThreadPoolExecutor.java:908) >>>>>>>> ... 1 more >>>>>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail >>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1==== >>>>>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root >>>>>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor >>>>>>>> transport >>>>>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail >>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully >>>>>>>> from >>>>>>>> [email protected] to [email protected] >>>>>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor >>>>>>>> for >>>>>>>> state root reached >>>>>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail >>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc==== >>>>>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root >>>>>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor >>>>>>>> transport >>>>>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail >>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully >>>>>>>> from >>>>>>>> [email protected] to [email protected] >>>>>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor >>>>>>>> for >>>>>>>> state root reached >>>>>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in >>>>>>>> RemoteDelivery.run() >>>>>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in >>>>>>>> RemoteDelivery.run() >>>>>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in >>>>>>>> RemoteDelivery.run() >>>>>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in >>>>>>>> RemoteDelivery.run() >>>>>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail >>>>>>>> while >>>>>>>> spooling Unable to dequeue next message >>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to >>>>>>>> dequeue next message; >>>>>>>> nested exception is: >>>>>>>> javax.jms.JMSException: PermGen space >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135) >>>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>>> Caused by: javax.jms.JMSException: PermGen space >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973) >>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown >>>>>>>> Source) >>>>>>>> at >>>>>>>> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>>>> at >>>>>>>> org.springframework.jms.connection.CachingConnectionFactory >>>>>>>> >>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344) >>>>>>>> at $Proxy61.createConsumer(Unknown Source) >>>>>>>> at >>>>>>>> >>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109) >>>>>>>> ... 2 more >>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote: >>>>>>>>> >>>>>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/). >>>>>>>>> If you go via the menu and open the dump file, it will propose you >>>>>>>>> (after a few minutes anlaysis) a report that will present you >>>>>>>>> "suspects". You can copy/paste this report on ml. >>>>>>>>> >>>>>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it >>>>>>>>> (not >>>>>>>>> sure it allows to analyze taken dumps, it more for live profiling). >>>>>>>>> >>>>>>>>> Tks, >>>>>>>>> - Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote: >>>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError >>>>>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding >>>>>>>>>> a easy >>>>>>>>>> way to analize dump file. Have you any suggestion? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nicola. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> Getting OOM can have various side effects, so I bet the >>>>>>>>>>> java.lang.VerifyError is a result of the OOM. >>>>>>>>>>> >>>>>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError >>>>>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump? >>>>>>>>>>> >>>>>>>>>>> If you have now time to analyze to find the "suspect", we can do >>>>>>>>>>> it for >>>>>>>>>>> you if you transfer us the dump (be aware that some objects can >>>>>>>>>>> contain >>>>>>>>>>> some user credentials, but that really doesn't interest >>>>>>>>>>> us). >>>>>>>>>>> >>>>>>>>>>> Tks, >>>>>>>>>>> - Eric >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> the problem persists, but now I have a new exception in the log: >>>>>>>>>>>> >>>>>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing >>>>>>>>>>>> mail while >>>>>>>>>>>> spooling PermGen space >>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing >>>>>>>>>>>> mail while >>>>>>>>>>>> spooling (class: javax/mail/MessagingException, method: >>>>>>>>>>>> superToString >>>>>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index >>>>>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, >>>>>>>>>>>> method: >>>>>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant >>>>>>>>>>>> pool >>>>>>>>>>>> index >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147) >>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing >>>>>>>>>>>> mail while >>>>>>>>>>>> spooling PermGen space >>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Nicola. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit >>>>>>>>>>>>> [1] of 20/5. >>>>>>>>>>>>> >>>>>>>>>>>>> If you still encounter the OOM, you can further analyse with >>>>>>>>>>>>> "eclipse >>>>>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you >>>>>>>>>>>>> the >>>>>>>>>>>>> objects that eat memory (launch james with standard Mx 512M, >>>>>>>>>>>>> otherwise >>>>>>>>>>>>> you will need more time to laod dump). >>>>>>>>>>>>> >>>>>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in >>>>>>>>>>>>> james-server-context.xml (see changes from [1]). >>>>>>>>>>>>> >>>>>>>>>>>>> Tks, >>>>>>>>>>>>> - Eric >>>>>>>>>>>>> >>>>>>>>>>>>> [1] http://s.apache.org/30x >>>>>>>>>>>>> [2] http://www.eclipse.org/mat/ >>>>>>>>>>>>> >>>>>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Eric, >>>>>>>>>>>>>> I'm using snapshot >>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400 >>>>>>>>>>>>>> >>>>>>>>>>>>>> now I'll try to install snapshot >>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407 >>>>>>>>>>>>>> and I'll add the complete debugging path in the startup script >>>>>>>>>>>>>> to >>>>>>>>>>>>>> further analyse the dump. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Many thanks for your help, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Nicola. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Tks for your answer. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So you've got more or less the same configuration as I have: >>>>>>>>>>>>>>> - maildir >>>>>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but >>>>>>>>>>>>>>> it's rejected >>>>>>>>>>>>>>> via smtp hook) >>>>>>>>>>>>>>> - not many users >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My config here is really stable after weeks (no oom, low >>>>>>>>>>>>>>> memory >>>>>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders >>>>>>>>>>>>>>> empty). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the >>>>>>>>>>>>>>> issue. >>>>>>>>>>>>>>> Would it come from recent >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/JAMES-1253 ? >>>>>>>>>>>>>>> btw, which version are you using ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The complete debugging path would be to add >>>>>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path >>>>>>>>>>>>>>> in the >>>>>>>>>>>>>>> startup script and to further analyse the dump: it will give >>>>>>>>>>>>>>> for sure >>>>>>>>>>>>>>> the cause of oom. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Eric >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have? >>>>>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> at this time we don't have very high mail traffic, more or >>>>>>>>>>>>>>>> less 500/600 >>>>>>>>>>>>>>>> mail per day. >>>>>>>>>>>>>>>> James is configured as smtp and imap servers and I think >>>>>>>>>>>>>>>> that most of >>>>>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active >>>>>>>>>>>>>>>> clients at the >>>>>>>>>>>>>>>> same time making requests to the James IMAP server). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This will allow us to better define the place to look at. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that >>>>>>>>>>>>>>>>> issue? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, it is in most cases. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is the log still showing messages after the OOM? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, please consider following log sequence: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 >>>>>>>>>>>>>>>> Got<tag>: 1125 >>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 >>>>>>>>>>>>>>>> Got<command>: >>>>>>>>>>>>>>>> noop >>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 >>>>>>>>>>>>>>>> executing hook >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb >>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 >>>>>>>>>>>>>>>> Got<tag>: A05974 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled >>>>>>>>>>>>>>>> mail will be >>>>>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" >>>>>>>>>>>>>>>> folder. If >>>>>>>>>>>>>>>> the number of mail rises too high then the problem occurs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Last thing: James is installed on a server machine whit >>>>>>>>>>>>>>>> following >>>>>>>>>>>>>>>> characteristics: >>>>>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel >>>>>>>>>>>>>>>> 2.6.37.6-0.5 >>>>>>>>>>>>>>>> java version: 1.6.0_25 >>>>>>>>>>>>>>>> RAM: 8Gb >>>>>>>>>>>>>>>> Architecture: x86_64 >>>>>>>>>>>>>>>> CPU(s): 6 >>>>>>>>>>>>>>>> On-line CPU(s) list: 0-5 >>>>>>>>>>>>>>>> Thread(s) per core: 1 >>>>>>>>>>>>>>>> Core(s) per socket: 6 >>>>>>>>>>>>>>>> CPU socket(s): 1 >>>>>>>>>>>>>>>> Vendor ID: AuthenticAMD >>>>>>>>>>>>>>>> CPU family: 16 >>>>>>>>>>>>>>>> Model: 8 >>>>>>>>>>>>>>>> Stepping: 1 >>>>>>>>>>>>>>>> CPU MHz: 2599.966 >>>>>>>>>>>>>>>> BogoMIPS: 5200.12 >>>>>>>>>>>>>>>> Virtualization: AMD-V >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for your help, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Nicola. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Tks, >>>>>>>>>>>>>>>>> - Eric >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine >>>>>>>>>>>>>>>>>> for one or >>>>>>>>>>>>>>>>>> two days until I start having memory problems and I need >>>>>>>>>>>>>>>>>> to restart >>>>>>>>>>>>>>>>>> James. >>>>>>>>>>>>>>>>>> The problems begin with a long series of messages such as >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | >>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker >>>>>>>>>>>>>>>>>> | Failed to register MBean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 >>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | >>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker >>>>>>>>>>>>>>>>>> | Failed to register MBean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 >>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | >>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker >>>>>>>>>>>>>>>>>> | Failed to register MBean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 >>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | >>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker >>>>>>>>>>>>>>>>>> | Failed to register MBean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and then >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process >>>>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to >>>>>>>>>>>>>>>>>> restart James. >>>>>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have >>>>>>>>>>>>>>>>>> -Xms512m -Xmx1024m) >>>>>>>>>>>>>>>>>> but the problem keeps coming back. >>>>>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem? >>>>>>>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Nicola. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>>> For additional commands, e-mail: [email protected] >>>>>>>> >>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [email protected] >>>>>>> For additional commands, e-mail: [email protected] >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
