Hi Eric, I could only of one change which could maybe improve things and so you don't see the OOM anymore. The patch introduced in revision r930727. This patch changed the behavoir of pop3servers RETR and TOP handler to not load the email in the memory while return it to the client.
Do you know which revision you used before ? Some more info would be interesting. Could you tell me after which POP3 command the Mailformed Packets appeared ? Thx, Norman 2010/4/6 Eric Charles <eric.char...@u-mangate.com>: > Hi Norman, > > james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day > in prod. > > jmap -dump:file=... (this morning and right now) shows > org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the > rest. > > However, I had to redefine all accounts as IMAP in thunderbird to be able to > read mails: > - the pop3 was working fine, and at a time-being, it hanged. > - Since there, all pop3 requests hang (thunderbird gives me a timeout after > 2 minutes), simply saying "downloading message 1 of ..." (nothing in the log > files) > - wireshark shows a correct conversation at the beginning, and afterwards > many "Malformed Packets". > Maybe I could restart james, but I prefer let him work to have a better view > on the OOM. > > Tks, > > Eric > > > On 04/02/2010 09:04 PM, Norman Maurer wrote: >> >> Hi Eric, >> >> I forgot to mention that one cause of the OOM could be the current >> usage of InMemoryMail. At the moment James load the whole MimeMessage >> into memory while processing the camel routes. This is really a bad >> practice in terms of performance and could maybe cause a OOM if you >> processing many big messages which are bigger (the sumary) as the >> avaible memory. >> Could this be the problem on your side ? >> >> I'm currently workin on storing the MimeMessage on an external storage >> and just stream in on demand.. So only the "meta-data" is stored in >> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin >> here. But I need a bit more time to see if its the way to go.. >> >> I'm also evolating the use of BlobMessages with ActiveMQ. >> >> Bye, >> Norman >> >> >> 2010/4/2 Norman Maurer<nor...@apache.org>: >> >>> >>> Hi Eric >>> >>> so at least the files not leak anymore, unfortunaly this was the less >>> critical problem. >>> About the errors in pop3server I will have a look and see what could >>> cause this. I'm 100 % sure it's not related to the OOM. >>> >>> Any more infos related to the OOM are welcome. >>> >>> Bye >>> Norman >>> >>> 2010/4/2, Eric Charles<eric.char...@u-mangate.com>: >>> >>>> >>>> nop. OutOfMemoryError is back. >>>> >>>> I will double check my server, jdk and environment config, but other >>>> java processes are running on the same machine without problem. >>>> I will also have a stress scenario on my local dev PC that runs fine and >>>> after, I will rerun it on my production server. >>>> Dump analysis and full profiling are also to do. >>>> Some stuff are difficult to simulate : See for example (occurs regulary) >>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> >>>> There are no more .m64 file in /tmp >>>> >>>> Tks, >>>> Eric >>>> >>>> >>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail >>>> Mail1270218298023-9820 from jewishnesses...@ecs-solar.com on >>>> 68.62.205.100 for [ric.char...@u-mangate.org, ric.char...@u-mangate.com] >>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms >>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail >>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/ >>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail >>>> Mail1270218307697-9822 from ours...@stahelin.com on 68.62.205.100 for >>>> [ric.char...@u-mangate.be] >>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail >>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/ >>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms >>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: cleanup took 7274 >>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: cleanup took 1141 >>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: cleanup took 58100 >>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase | >>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took >>>> 52100 ms >>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail >>>> Mail1270218355606-9824 from inexperienc...@toutgratuit.com on >>>> 213.163.116.78 for [angelo.bid...@u-mangate.net] >>>> WARN 16:36:27,919 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination >>>> 'processor.local-address-error' - trying to recover. Cause: Java heap >>>> space >>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> WARN 16:44:23,231 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination 'processor.spam' >>>> - trying to recover. Cause: Java heap space >>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an >>>> exception trying to create a JMSException for >>>> java.lang.OutOfMemoryError: Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an >>>> valid Object but class java.lang.String >>>> WARN 17:09:11,055 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination >>>> 'processor.local-address-error' - trying to recover. Cause: Java heap >>>> space >>>> ERROR 17:09:35,017 | >>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer >>>> >>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000] >>>> could not poll endpoint: >>>> >>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000 >>>> caused by: Java heap space >>>> java.lang.OutOfMemoryError: Java heap space >>>> WARN 17:09:55,361 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination 'processor.spam' >>>> - trying to recover. Cause: Java heap space >>>> WARN 17:09:57,557 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination 'processor.root' >>>> - trying to recover. Cause: Java heap space >>>> WARN 17:11:00,695 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination >>>> 'processor.local-address-error' - trying to recover. Cause: Java heap >>>> space >>>> WARN 17:22:19,706 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination >>>> 'processor.bounces' - trying to recover. Cause: Java heap space >>>> WARN 17:22:42,422 | >>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup >>>> of JMS message listener invoker failed for destination >>>> 'processor.transport' - trying to recover. Cause: Java heap space >>>> >>>> >>>> On 04/02/2010 09:00 AM, Norman Maurer wrote: >>>> >>>>> >>>>> Hi Eric, >>>>> >>>>> this sounds promising.... Looking forward for your next report. >>>>> >>>>> Bye >>>>> Norman >>>>> >>>>> 2010/4/2, Eric Charles<eric.char...@u-mangate.com>: >>>>> >>>>> >>>>>> >>>>>> James survived the night. >>>>>> 300MB (exactly the same as when it started). >>>>>> No m64 file in tmp. >>>>>> >>>>>> I will post another update this evening. >>>>>> >>>>>> Tks, >>>>>> >>>>>> Eric >>>>>> >>>>>> >>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote: >>>>>> >>>>>> >>>>>>> >>>>>>> Yeah thats right.. just checout server/trunk. >>>>>>> >>>>>>> I'm looking forward to hear your feedback ;) >>>>>>> >>>>>>> Bye, >>>>>>> Norman >>>>>>> >>>>>>> >>>>>>> 2010/4/1 Eric Charles<eric.char...@u-mangate.com>: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Hello Norman, >>>>>>>> Done :) >>>>>>>> Tks, Eric >>>>>>>> >>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am >>>>>>>> using. >>>>>>>> PS2 : I only checkout >>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk >>>>>>>> and >>>>>>>> not the other projects. From what I read from the poms, snapshot >>>>>>>> version >>>>>>>> of >>>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository >>>>>>>> and >>>>>>>> I >>>>>>>> don't have to rebuild those projects. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Eric, >>>>>>>>> >>>>>>>>> its me again ;) Could you please svn up to latest trunk and try >>>>>>>>> again >>>>>>>>> ? >>>>>>>>> >>>>>>>>> Thx, >>>>>>>>> Norman >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> 2010/4/1 Norman Maurer<norman.mau...@googlemail.com>: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Eric, >>>>>>>>>> >>>>>>>>>> I did a stress test in the last few hours (running in the >>>>>>>>>> background >>>>>>>>>> here) and I was not able to get a OOM yet with >>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the >>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and >>>>>>>>>> 3000 >>>>>>>>>> with 5 mb. >>>>>>>>>> >>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then >>>>>>>>>> 700m. >>>>>>>>>> >>>>>>>>>> Bye. >>>>>>>>>> Norman >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>> >>>>> >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>>> >>>> >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
