2) Launching james, memory usage is 300MB. This seems normal and holding the traffic well. At a certain time, memory jumps to 3GB (I gave much memory via -Xmx).
Maybe a certain condition produces this.
Debugging in eclipse shows me that many threads are created by camel. I also had the impression that some threads were falling and other ones were recreated. If this is the case, is it normal. I'm going to look at activemq/camel (I don't know them really), and also still read the http://activemq.apache.org/javalangoutofmemory.html, http://activemq.apache.org/my-producer-blocks.html,...

I have the impression that everything is working fine, but at a certain moment, there is a condition that makes the memory usage rise extensively (not a linear growth, but really a stair).

Bye,
Eric


On 04/01/2010 09:24 AM, Norman Maurer wrote:
Hi Eric,

thx again for your tests...

So I think we have two independent problems here:

1) The .m64 files which are not deleted always:

My guess here is that there is something wrong with processing Mail
objects with state Null. I will investigate futher.. I also found a
possible problem with .m64 files which get not deleted when a
Exception is throws while processing the camel routes. I will patch
this soon.

2) The OOM

My guess here is that ActiveMQ is not freeing up memory. Not sure why yet

I think 2) is the more criticial problem and should get higher
priority to get fixed..

Bye,
Norman



2010/4/1 Eric Charles<eric.char...@u-mangate.com>:
Hi Norman,

Tks for the smpt-source pointer.
I looked at it, but finally, I think I will develop those little classes to
have more control on it.
I will put it on github.

I launched latest trunk james yesterday evening.
During the night, 3 hours after the launch, the following occured:

KahDB slower and slower...

INFO  23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
KahaDB access: cleanup took 2330
INFO  23:38:04,591 | james.mailetcontext | Error while storing mail.
org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
parsed.;
  nested exception is:
    org.apache.james.imap.mailbox.StorageException: failed. Transaction
commit failed.;
  nested exception is:
<openjpa-1.2.1-r752877:753278 fatal store error>
org.apache.openjpa.persistence.RollbackException: The transaction has been
rolled back.  See the nested exceptions for details on the errors that
occurred.
    at
org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)

    at
org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
    ... 55 more
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
mediaType, subType, textu
alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260,
(byte[]) [...@7a00e70c, (long) 47887, (String) text, (String) html, (long)
876]} [code=0, state=XJ001]
    at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
    at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
    at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
    at
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)

So still a OOM exception that was shown by yet-another-component (in this
case, the StoreMailbox).


There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
job).
All 4 files were 0 bytes.

I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
memory).

With the previous parameters ( -Xmx512m), the process was taking the whole
512MB.

Tks,
Eric


On 03/31/2010 08:26 PM, Norman Maurer wrote:
Hi eric,

thx for all your help, maybe smtp-source ( which is included in
postfix ) can help you with stress testing.

Bye
Norman

2010/3/31, Eric Charles<eric.char...@u-mangate.com>:

Hi Norman,

There are not so much .m64 files, so even it it shows that a mail
processing didn't reach the end and may have cause a leak, I don't think
it would make crash my process so quick (between 10 and 24 hours).

I quickly analysed a few dumps last week : there is always a class that
took 30/40% of the memory. Sometimes a activemq class, sometimes not
(don't remember the details).
Also notable, the stack trace (the place where the OOM gives problems)
varies.
Today stack was:
org.apache.camel.RuntimeCamelException:
org.apache.camel.CamelExchangeException: Error processing Exchange.
Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
responseRequired = true, messageId =
ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
originalTransactionId = null, producerId =
ID:srv001-51032-1270011735798-2:0:24:1, destination =
queue://processor.root, transactionId = null, expiration = 0, timestamp
= 1270014325685, arrival = 0, brokerInTime = 1270014326893,
brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
persistent = true, type = null, priority = 4, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID =
null, content = org.apache.activemq.util.byteseque...@69b568d0,
marshalledProperties = null, dataStructure = null, redeliveryCounter =
0, size = 10718, properties = null, readOnlyProperties = true,
readOnlyBody = true, droppable = false}]. Caused by:
[java.lang.OutOfMemoryError - Java heap space]
      at

org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
      at

org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
...



I also think the tmp files are not the main cause.

I have now james running in eclipse and I made already made a little
trip in the code. Quite impressive since last time I looked at it.
I will try to stress james with a small smtp/pop3 client (I used
postage, but a simple class with commons-net behind) may be easier and
see what happens (eventually with eclipse profiler).

It will be for this weekend for me.

Bye,

Eric


On 03/31/2010 07:11 PM, Norman Maurer wrote:

Hi Eric,

thx for keeping us in the loop... I'm still not sure why the .m64
files are still in the tmp folder sometimes.. But I suspect the files
are not the cause of the OOM.

I didn't get a OOM since yesterday morning (the time I deployed
current trunk version), but just found to new .m64 files..

So I'm still searching for the real cause. If nothing helps I will
need to use a profiler to find th leak.

Bye,
Norman

2010/3/31 Eric Charles<eric.char...@u-mangate.com>:


Hi Norman,

I had defined the Null mailet

<mailet match="HostIsLocal" class="Null">
<processor>     local-address-error</processor>
<notice>550 - Requested action not taken: no such user here</notice>
</mailet>

but now, I use the standard config

<mailet match="HostIsLocal" class="ToProcessor">
<processor>     local-address-error</processor>
<notice>550 - Requested action not taken: no such user here</notice>
</mailet>

I still have the OOM.

I will now deploy a fresh svn with your last commits and enable the
ValidRcptHandler.
I keep you posted with the result,

Tks,
Eric


On 03/30/2010 06:49 AM, Norman Maurer wrote:


Hi Eric,

you said all the files are related to address-errors , could you show
me your address error processor config?
Does the Problem still exist when you enable the ValidRcptHandler in
the smtpserver.xml file?

Thx
Norman

2010/3/30, Eric Charles<eric.char...@u-mangate.com>:



Oops, no, the files are still there (only unkn...@known.com).
Eric


On 03/29/2010 10:16 PM, Eric Charles wrote:



Hi Norman,

I just deployed your new commit with the new camel DisposeProcess.
Good news : I don't see anymore the m64 file in tmp (well I see 1
file, on the second after, it is no more there, so the dispose works
as it should).

I keep you posted with our eventual future OOM.

Tks,
Eric



On 03/29/2010 09:18 PM, Eric Charles wrote:



Hi Norman,

The .m64 are all to "unkown user" but to "well known domain" (so
for
<unkn...@known.com>).
They are from various size (with and without attachment).
They are well formed (I can open the downloaded file with
thunderbird)

However, when I sent a mail to unkn...@known.com, I don't see it in
the /tmp

Tks,
Eric


On 03/29/2010 08:43 PM, Norman Maurer wrote:



Hi Eric,

sure.. we have to find the OOM cause. What would be interesting,
could
you check somehow if the .m64 files are files which are related to
successfully delivered mail ?

Thx,
Norman


2010/3/29 Eric Charles<eric.char...@u-mangate.com>:



Norman,

Done. Now, we have to wait...

I saw some created *.m64 that were removed.
But there are other ones that remains in /tmp.

I sometimes run a jmap (java memory map) to produce a heap dump
and
analyse
it.
At the beginning, everything seems ok, and often, when I come
back,
the OOM
has already occured.

I changed the -Xmx512m to -Xmx256m to have a quicker exception
(if
any,
let's hope not).

The *.m64 are a clue and we should ensure that they are removed
in
all
circumstances.
This may be the easy part, as we have some visible clues (the
files).
We should also ensure after that there are no other causes to the
leaks.

Tks,
Eric


On 03/29/2010 08:00 PM, Norman Maurer wrote:



Hi Eric,

I found the cause for the not deleted temporary files. Hopefully
this
is the cause of the OOM. Could try to svn up and run again ?

Thx,
Norman


2010/3/29 Norman Maurer<norman.mau...@googlemail.com>:




Hi Eric,

thx for the report. I see exact the same problem today here..
(The
OOM). Didn't notice the files in the tmp folder, but I think
thats
a
good pointer. I will try to debug the problem later or
tomorrow.
But I
suspect you are right about the tmp files and jms producers..
Did
you
run a kill -3 pid to see what threads are active etc ?

Thx,
Norman

Ps: Patches are welcome :)

2010/3/29 Eric Charles<eric.char...@u-mangate.com>:




Hi,

The last two weeks, I deployed various trunk snapshots.
After james running less than 1 day, I always ran into a
OutOfMemory
Exception.

I analysed the logs (for example STACK TRACE 1 in annex), and
always
found
Camel complaining when trying to deliver on JMS queue.
I tried to adapt the ActiveMQ configuration based on
http://activemq.apache.org/javalangoutofmemory.html but
nothing
helped.

I also analyzed various heap dump :sometimes, 65% was used by
org.apache.mina.transport.socket.nio.NioSocketSession,
sometimes
by a
activemq class,...

















---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org






---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org






---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org

Reply via email to