Hi Eric,
could you try if this helps...
In spring-beans.xml replace the jmsConnectionFactory block with:
<!-- jms connection pooling -->
<bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnec
tionFactory" destroy-method="stop">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>vm://localhost?broker.useJmx=false&jms.prefetchPolicy.all
=50</value>
</property>
</bean>
</property>
</bean>
Bye,
Norman
2010/4/1 Eric Charles <eric.char...@u-mangate.com>:
> 2) Launching james, memory usage is 300MB. This seems normal and holding the
> traffic well. At a certain time, memory jumps to 3GB (I gave much memory via
> -Xmx).
> Maybe a certain condition produces this.
> Debugging in eclipse shows me that many threads are created by camel. I also
> had the impression that some threads were falling and other ones were
> recreated. If this is the case, is it normal.
> I'm going to look at activemq/camel (I don't know them really), and also
> still read the http://activemq.apache.org/javalangoutofmemory.html,
> http://activemq.apache.org/my-producer-blocks.html,...
>
> I have the impression that everything is working fine, but at a certain
> moment, there is a condition that makes the memory usage rise extensively
> (not a linear growth, but really a stair).
>
> Bye,
> Eric
>
>
> On 04/01/2010 09:24 AM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> thx again for your tests...
>>
>> So I think we have two independent problems here:
>>
>> 1) The .m64 files which are not deleted always:
>>
>> My guess here is that there is something wrong with processing Mail
>> objects with state Null. I will investigate futher.. I also found a
>> possible problem with .m64 files which get not deleted when a
>> Exception is throws while processing the camel routes. I will patch
>> this soon.
>>
>> 2) The OOM
>>
>> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>>
>> I think 2) is the more criticial problem and should get higher
>> priority to get fixed..
>>
>> Bye,
>> Norman
>>
>>
>>
>> 2010/4/1 Eric Charles<eric.char...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> Tks for the smpt-source pointer.
>>> I looked at it, but finally, I think I will develop those little classes
>>> to
>>> have more control on it.
>>> I will put it on github.
>>>
>>> I launched latest trunk james yesterday evening.
>>> During the night, 3 hours after the launch, the following occured:
>>>
>>> KahDB slower and slower...
>>>
>>> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow
>>> KahaDB access: cleanup took 2330
>>> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
>>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>>> parsed.;
>>> nested exception is:
>>> org.apache.james.imap.mailbox.StorageException: failed. Transaction
>>> commit failed.;
>>> nested exception is:
>>> <openjpa-1.2.1-r752877:753278 fatal store error>
>>> org.apache.openjpa.persistence.RollbackException: The transaction has
>>> been
>>> rolled back. See the nested exceptions for details on the errors that
>>> occurred.
>>> at
>>>
>>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>>
>>> at
>>>
>>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>>> ... 55 more
>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>> contentOctets,
>>> mediaType, subType, textu
>>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int)
>>> 1260,
>>> (byte[]) [...@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>>> 876]} [code=0, state=XJ001]
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>>
>>> So still a OOM exception that was shown by yet-another-component (in this
>>> case, the StoreMailbox).
>>>
>>>
>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>> job).
>>> All 4 files were 0 bytes.
>>>
>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>> max
>>> memory).
>>>
>>> With the previous parameters ( -Xmx512m), the process was taking the
>>> whole
>>> 512MB.
>>>
>>> Tks,
>>> Eric
>>>
>>>
>>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi eric,
>>>>
>>>> thx for all your help, maybe smtp-source ( which is included in
>>>> postfix ) can help you with stress testing.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/3/31, Eric Charles<eric.char...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> There are not so much .m64 files, so even it it shows that a mail
>>>>> processing didn't reach the end and may have cause a leak, I don't
>>>>> think
>>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>>
>>>>> I quickly analysed a few dumps last week : there is always a class that
>>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>>>> (don't remember the details).
>>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>>> varies.
>>>>> Today stack was:
>>>>> org.apache.camel.RuntimeCamelException:
>>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>>> responseRequired = true, messageId =
>>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>>>> originalTransactionId = null, producerId =
>>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>>>>> =
>>>>> null, content = org.apache.activemq.util.byteseque...@69b568d0,
>>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>>> ...
>>>>>
>>>>>
>>>>>
>>>>> I also think the tmp files are not the main cause.
>>>>>
>>>>> I have now james running in eclipse and I made already made a little
>>>>> trip in the code. Quite impressive since last time I looked at it.
>>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>>> postage, but a simple class with commons-net behind) may be easier and
>>>>> see what happens (eventually with eclipse profiler).
>>>>>
>>>>> It will be for this weekend for me.
>>>>>
>>>>> Bye,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>>>> are not the cause of the OOM.
>>>>>>
>>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>>> current trunk version), but just found to new .m64 files..
>>>>>>
>>>>>> So I'm still searching for the real cause. If nothing helps I will
>>>>>> need to use a profiler to find th leak.
>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>> 2010/3/31 Eric Charles<eric.char...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> I had defined the Null mailet
>>>>>>>
>>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>>> <processor> local-address-error</processor>
>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>> </mailet>
>>>>>>>
>>>>>>> but now, I use the standard config
>>>>>>>
>>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>>> <processor> local-address-error</processor>
>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>> </mailet>
>>>>>>>
>>>>>>> I still have the OOM.
>>>>>>>
>>>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>>>> ValidRcptHandler.
>>>>>>> I keep you posted with the result,
>>>>>>>
>>>>>>> Tks,
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> you said all the files are related to address-errors , could you
>>>>>>>> show
>>>>>>>> me your address error processor config?
>>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>>>>> the smtpserver.xml file?
>>>>>>>>
>>>>>>>> Thx
>>>>>>>> Norman
>>>>>>>>
>>>>>>>> 2010/3/30, Eric Charles<eric.char...@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Oops, no, the files are still there (only unkn...@known.com).
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Norman,
>>>>>>>>>>
>>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>>>>> file, on the second after, it is no more there, so the dispose
>>>>>>>>>> works
>>>>>>>>>> as it should).
>>>>>>>>>>
>>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>
>>>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so
>>>>>>>>>>> for
>>>>>>>>>>> <unkn...@known.com>).
>>>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>>>>> thunderbird)
>>>>>>>>>>>
>>>>>>>>>>> However, when I sent a mail to unkn...@known.com, I don't see it
>>>>>>>>>>> in
>>>>>>>>>>> the /tmp
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>>>>> could
>>>>>>>>>>>> you check somehow if the .m64 files are files which are related
>>>>>>>>>>>> to
>>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.char...@u-mangate.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Norman,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump
>>>>>>>>>>>>> and
>>>>>>>>>>>>> analyse
>>>>>>>>>>>>> it.
>>>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come
>>>>>>>>>>>>> back,
>>>>>>>>>>>>> the OOM
>>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception
>>>>>>>>>>>>> (if
>>>>>>>>>>>>> any,
>>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>>
>>>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed
>>>>>>>>>>>>> in
>>>>>>>>>>>>> all
>>>>>>>>>>>>> circumstances.
>>>>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>>>>> files).
>>>>>>>>>>>>> We should also ensure after that there are no other causes to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> leaks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I found the cause for the not deleted temporary files.
>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/3/29 Norman Maurer<norman.mau...@googlemail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> good pointer. I will try to debug the problem later or
>>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers..
>>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.char...@u-mangate.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex),
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but
>>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
>> For additional commands, e-mail: server-dev-h...@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
> For additional commands, e-mail: server-dev-h...@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
