Hi there,
Today is an excellent step towards a more performant imap channel.
With Norman's last commit, i re-stressed and analyzed the memory dumps
(with eclipse mat), and everything is fine now: very low memory
consumption per mailbox session.
The "leaking" listeners (when other mailbox where selected) were the
cause, and now they are "removed" (content set to null).
Just before upcoming imap 0.2-M1 : that's really good news!! ;)
Tks,
Eric
On 14/10/2010 14:09, Eric Charles wrote:
Total Retained Heap: 341.033.584
where top is OpenJPAMailboxManager: 266.079.320 which references
266.068.064 of DelegatingMailboxListeners.
In DelegatingMailboxListerners, there is the Map<MailboxPath,
List<MailboxListener>> listeners.
I can see there the 25 map entries, where
- first one is 105.017.544
- second one is 71.025
-...
for the total of 266.068.064
The first map entry for the MailboxPath being of large subfolder of my
inbox has
- 301 instances of UidToMsnConverter in a list for 87.229.424
- 200 instances of UidToMsnConverter in a list (which is it self in a
map) for 17.788.032
Each UidToMsnConverter contains 2 TreeMap (msnToUid and uidToMsn),
each having 1.037.984.
Have to replay it, but could we have one instance of UidToMsnConverter
per session to spare some memory?
Tks,
Eric
On 14/10/2010 13:44, Norman wrote:
Hi Eric,
how big was the memory usage of the Map which holds the MailboxPath ?
Just to get an idea if we need to improve it.
Bye,
Norman
Am 14.10.2010 13:35, schrieb Eric Charles:
Hi Norman,
Yes, it solved oom atm.
I was not able to reproduce it while fetching mails from 2 clients.
However, I saw very fast fetching at start, slowing down at the end.
So, I fear that launching many clients in // to my mailbox would
still slow it, and probably give oom (but it's true that I use the
default -Xmx512M).
I took a mem dump, and at first look, jpaheader,... object do not
eat the memory anymore.
The largest memory usage is OpenJpaManager which references a Map
with many MailboxPath.
Yes, gc logs should be activated, and we should also find the best
configs (there's many one).
I also would like to test the G1 gc that seems more stable now than
it was last year.
But i will stop monitoring/analyzing gc/memory for now to
concentrate on M1 release.
We should take this between M1 and M2, especially with automated
stress tests on all protocols (stmp/imap/pop3/...).
Tks,
Eric
On 14/10/2010 13:12, Norman wrote:
Hi Eric,
so the patch solves the OOM for you ? Did you have a look at the
heap (via jconsole;) ) while do the action ? Would be interesting
to see how the GC kicks in..
And yes I think we can have it in M1 ;)
Bye
Norman
Am 14.10.2010 12:10, schrieb Eric Charles:
Hi Norman,
I tested with the patch you sent me off-ml, and it rebuilding
index on a large index goes 10 ways faster :)
I still didn't succeed to get the oom synchronizing.
Do you think we could have this patch in the coming imap 0.2-M1
release?
For later releases, we could still talk about ways to enhance the
FETCH performance (full streaming solution for example).
Tks,
Eric
On 12/10/2010 10:27, Eric Charles wrote:
- read "raw mail" instead of "raw content"
- to have added value, "streaming from store to socket via
listener" goes with "store raw mail".
On 12/10/2010 10:10, Eric Charles wrote:
Hi Norman,
Yes, Iterator<MessageResult> getMessages(MessageRange set, ...)
is really the place where we have to act to reduce the memory
consumption.
I also though about some ways to better scale.
I didn't really think to a batch way, but this can help, even if
we will have to go to the repository (database for example) 10
times in your example.
Also I'm not sure the gc will act before the end of the batch
processing (depends on the gc config), so you could still have
memory growing very fast to serve a huge fetch, without being
really garbaged.
But this could be a quite quick fix (if easy to implement ?)
I thought here on "streaming" and "raw content".
- Streaming: I worked on a project where the http socket was
directly streamed to the jdbc driver for file upload, and the
inverse for file download. For mail, upload is more smtp and is
managed by jms queue (with eventual claim-check pattern if
needed). The download part is imap, and we load everything in
memory before writing the response. We could imagine some kind
of listener that would be notified of each mail reading (reading
done via stream). The listener would write the response via
stream also, without storing it in memory.
- "raw content". The store API forces to work with
MailboxMembership that clearly isolates the headers from the
content,... This is good to map the IMAP RFC that allows a
client to ask for headers only for example. With the current
API, we "impose" the store to structure the mails in
content/header/... and if the store can not structure the mail
(for example MailDir), we impose it to parse it, even if we have
to return the raw content in case of a classical plain FETCH
command... I was wondering if we could not add a raw content
storage obligation for all stores, and if we need to return the
raw content, simply use it. This could play nice with the stream
option above.
But yes, the batch approach is quicker to implement.
Tks,
Eric
On 11/10/2010 19:28, Norman Maurer wrote:
Hi there,
I tried to get my head around the cause of the OOM and I think
I found
a good solution. Let me outline the idea...
The MessageManager interface expose many methods which expose
kinds of
Iterator<...> objects. So the idea is to build up some "batch
retrieve
Iterator" which then retrieve the needed data in batches. So
the GC
has a chance to kick in and free up resources. So here is an
example..
When call MessageManager.getMessages(...) (which returns
Iterator<MessageResult> ) and use a MessageRange of 1:1000 we
would
fetch the MessagResult 1:100 as starting point. Then wait till
Iterator.hasNext() or Iterator.next() is called and we have no
MessageResult left we would fetch 101:200 and so on..
I think this should work and will be much more efficient in
terms of
memory. All this could get done in the "store" implementations and
could be configurable. Like use 100 as default batch count and
be able
to set it via a setter.
WDYT ?
Bye,
Norman
2010/10/11 Norman Maurer<nor...@apache.org>:
Hi Eric,
Did you also have a look with wireshark what the exact command
and
argument was which triggered the OOM?
Thx
Norman
2010/10/11, Eric Charles<e...@apache.org>:
Hi Norman,
There were 2 main problems:
1. The amq one which is now resolved tks to your last commit
2. James no more responding on imap which is always caused by
OOM (I
missed some log the first time).
For the second one, analysis of memory dump shows oom comes
from huge
usage of memory due to loading of message, headers,... (in
case of
10.000 message fetch for example).
I don't benefit from Lob streaming on derby database, but it
won't help
much because jpaheader for example also take much memory.
Tks,
Eric
On 11/10/2010 13:10, Norman Maurer wrote:
Ok 4/5 is fixed now... Just to keep you updated..
Bye.
Norman
2010/10/11 Norman Maurer<nor...@apache.org>:
Ok at least you can reproduce it, thats good ;) Did you
take a thread
dump ?
Bye,
Norman
2010/10/11 Eric Charles<e...@apache.org>:
It's the same with latest thunderbird
I restarted disabling 'Check for new messages on startup
on all my
accounts.
If I go quickly from one folder to another, I fall back in
the endless
'downloading'/'indexing'...
However, if I quietly click on 'Get Mail' folder per
folder, it's ok.
I think we are still with Bug 1 (Bug 2 and 3 should be
resolved if 1 is
resolved) for IMAP, fetching simultaneously some folders.
Bug 4 is for amq.
Tks,
Eric
On 10/10/2010 20:03, Eric Charles wrote:
I tried to resync thunderbird without clicking on any
folder.
Still the same behaviour : "downloading xxx on yyy", www
on zzz,...
Wireshark tells me more: I never saw such red/black lines
in the tcp
stream (one red/black on every 5/10 tcp packet: "segment
lost").
1783 8.626604 91.183.38.48 192.168.1.12
IMAP [TCP
Previous
segment lost] Response:
ss.properties?rev=1005079&r1=1005078&r2=1005079&view=diff
I was wondering if my cable was right:
- tested plain http via cable: wireshark is green.
- tested thunderbird/james via wifi : same black/red
lines in
wireshark.
I have saved the dump and will analyze further tomorrow,
but a tcp
conversation selected from a "segment lost" seems ok.
So for now (this may change), I think we have:
1. A client is in a stage that causes the "segment lost"
tcp errors ==>
Bug 1
2. Client/server conversation loops endless ==> Bug 2
3.1. James finally hangs ==> Bug 3
3.2. James finally gets oom ==> Bug 3
4. Manual stop is needed.
5. After manual stop in state 3.1 or 3.2, there's a activemq
java.io.EOFException: Chunk stream does not exist at
page: 0 ==> Bug 4
So 4 bugs ?
I will upgrade my thunderbird 3.0.3 on linux to the
latest version and
see
if bug 1 is not resolved.
Bug 4 may be resolved with 5.4.1 and latest commits for
the james stop
procedure.
Tks,
Eric
On 10/10/2010 18:31, Eric Charles wrote:
Hi,
I have on James 3 (trunk of 2 week ago) my INBOX with 10
subfolders,
some
of these subfolders having more than 10.000 mails.
I mainly use a PC, so the IMAP sync is done regulary
along the day.
I also have another PC I synchronize once a week.
During the IMAP sync of that PC, I selected randomly
some subfolders
and
saw (this occured twice, but not always...):
- Thunderbird syncs well during a some minutes (10?)
- After, Thunderbird begins to say "downloading xx of yy
mails"..
.when
yy is reached, he says "downloading ww of zz" where zz
is a little
greater
than yy.
- I wait, wait, and finally have timeout, and the mails
are no more
viewable in thunderbird.
James is stucked.
The first time I had OOM (I think), today, I had no OOM,
but James was
no
more reachable via IMAP, though accepting mails via SMTP.
I stopped, and when restarting, I had the following
exception (James
was
not usable anymore):
INFO 18:16:37,646 |
org.apache.activemq.store.kahadb.plist.PListStore
|
PListStore:activemq-data/localhost/tmp_storage started
INFO 18:16:37,648 |
org.apache.activemq.broker.BrokerService | Using
Persistence Adapter:
KahaDBPersistenceAdapter[activemq-data/localhost/KahaDB]
INFO 18:16:38,248 |
org.apache.activemq.store.kahadb.plist.PListStore
|
PListStore:../data/localhost/tmp_storage started
ERROR 18:16:38,301 |
org.apache.activemq.broker.BrokerService | Failed
to
start ActiveMQ JMS Message Broker. Reason:
java.io.EOFException: Chunk
stream does not exist at page: 0
java.io.EOFException: Chunk stream does not exist at
page: 0
at
org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:454)
at
org.apache.kahadb.page.Transaction$2.<init>(Transaction.java:431)
at
org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)
at
org.apache.kahadb.page.Transaction.load(Transaction.java:404)
at
org.apache.kahadb.page.Transaction.load(Transaction.java:361)
at
org.apache.activemq.store.kahadb.MessageDatabase$1.execute(MessageDatabase.java:243)
at
org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:230)
at
org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:309)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:353)
at
org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:217)
at
org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:178)
Sounds l ike
https://issues.apache.org/activemq/browse/AMQ-2935.
To solve it, I had to remove the activemq-data directory
(btw, 2 weeks
ago was activemq 5.4.0 with 2 brokers started and
activemq-data in bin
directory).
I made a test to restart from scratch my account in
thunderbird, and
it
was OK.
Is it because it does a incremental sync and I select
different
folders
(just to make things complicated :) ) during the download ?
Anyway, it is not easy to reproduce.
Activemq 5.4.1. may be worth to try, but I'm not sure it
the the
cause...
Tks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail:
server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:
server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail:
server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:
server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail:
server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:
server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:
server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail:
server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
