Re: JAMES OOM, maybe the cause of MINA ?
Hi, I finally made some tests with mina-rc2 trying to reproduce the oom. I didn't succeed to reproduce it when stressing james with normal mails (well-formed,...). I would tend to follow the pointer Norman gave about the infinite loop in the exceptionCaught() method (writing on the session recause and exception,...) http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014&view=markup I think that some sockets behaving badly may be the cause and tried to simulate this with long-time opened connection, with garbage being sent on the sockets,... without success (no exception). I will now roolback to the previous version of SMTPIoHandler with james not closing the mina session, and see what I can find. If you've got some idea of patterns, datas,... to send on the socket to make mina throw an exception, just post it. Tks, Eric On 04/13/2010 07:23 PM, Eric Charles wrote: Hi Ashish, James in running with netty since 2 days. I plan to let it run until this Friday (if no problem). Thereafter, I will go back to mina. The logs I've got are not helpfull and show nothing, except a sudden java OOM. I may try this weekend to find a reproducible way to have the oom. Tks, Eric On 04/13/2010 10:47 AM, Norman Maurer wrote: We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashish: Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, James in running with netty since 2 days. I plan to let it run until this Friday (if no problem). Thereafter, I will go back to mina. The logs I've got are not helpfull and show nothing, except a sudden java OOM. I may try this weekend to find a reproducible way to have the oom. Tks, Eric On 04/13/2010 10:47 AM, Norman Maurer wrote: We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashish: Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashish : > Is the OOM fixed after the change? > > thanks > ashish > > On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer > wrote: >> Hi Ashish, >> >> I bet its the loop I mention before.. This would make sense when >> looking at the growing queued writes. I commited a change to trunk >> which close the session after write the data to the client. We will >> see if this helps.. >> >> Thx, >> Norman > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer wrote: > Hi Ashish, > > I bet its the loop I mention before.. This would make sense when > looking at the growing queued writes. I commited a change to trunk > which close the session after write the data to the client. We will > see if this helps.. > > Thx, > Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Forgot to include JAMES list.. 2010/4/12 Norman Maurer : > Hi Asish, > > yeah in our case this is fine. But I think even if this fix the > problem MINA should at least try to "break" such loops. > > Bye, > Norman > > > 2010/4/12 Ashish : >> On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer >> wrote: >>> Hi Ashish, >>> >>> I bet its the loop I mention before.. This would make sense when >>> looking at the growing queued writes. I commited a change to trunk >>> which close the session after write the data to the client. We will >>> see if this helps.. >> >> I hope it solves the problem :) >> Closing the session means disconnecting the client. If the situation >> wants this its fine, else need to dig a bit more for solution :) >> >> If the problem is solved, I consider its a good start for the week :) >> >>> >>> Thx, >>> Norman >> > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer wrote: > Hi Ashish, > > I bet its the loop I mention before.. This would make sense when > looking at the growing queued writes. I commited a change to trunk > which close the session after write the data to the client. We will > see if this helps.. I hope it solves the problem :) Closing the session means disconnecting the client. If the situation wants this its fine, else need to dig a bit more for solution :) If the problem is solved, I consider its a good start for the week :) > > Thx, > Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Sun, Apr 11, 2010 at 8:57 PM, Norman Maurer wrote: > Hi Ashish, > > after thinkin more about the whole stuff I wonder if the exception > handling in SMTP could sometimes be an endless loop. In the handler we > write a message back to the client on exception. But what happens if > this throw a new exception ? Could it so loop forever ? I am not sure, but it could be possible. > > http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014&view=markup > > Maybe we should just close the connection.. hmm.. its brute force. I am not convinced that we should do this, but not sure as I don't have a complete picture as such. If the exception is coz of packet decoding, ignore the packet and move on.. Its also driven by the protocol as such. You may need to respond back to the client with some error code etc. Eric: Please mark a copy to mina dev ML as well, else I don't get your mails :) The analysis is a session is in hanging state, whose queue is building up. Heap dump are not going to help any more. Can you trace down what the hack is going on with that session. I can't get much out of heap dump and unfortunately not able to run it on laptop. jconsole simply refuse to detect the James process :( So the key lies in analyzing 1. Why is this session hanging, its simply not writing the data back to the client. There is a while before the session is closed and session.isConnected() returns false. Eric: can you upload the logs as well, if possible :) Anyone else has some idea's here? thanks ashish > > Bye > Norman > > 2010/4/9 Eric Charles : >> Hi Ashish, >> >> I am the user who has many OOM with the trunk and feed Norman with my issues >> :) >> >> 1. >> Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is >> the release we normally use in james and give the issue after a certain >> period of time. I mean, the server can stay working correctly 2 days or >> crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due >> to a peak in memory usage. T >> To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, >> we see a direct crash. I've uploaded such a dump on >> http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force >> refresh, I setup the web server in a hurry). You will see there recursive >> ConcurrentLinkedQueue. >> >> 2. >> As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works >> while the line is flat (mail are received, spooled, delivered,...). On a few >> seconds, it peaks and of course, nothing works anymore. So, the messages >> don't remain in the spool. I made tests with quite huge messages, and they >> are delivered very fast as soon as they arrive in James. >> >> Don't hesitate to ask more questions or propose additional tests. >> >> Many tks in advance, >> >> Eric >> >> >> On 04/09/2010 12:40 PM, Ashish wrote: >>> >>> Norman, >>> >>> Couple of more queries >>> >>> 1. The heap dump uses circularqueue class, so seem to be taken for an >>> earlier trunk snapshot. Is my take correct? >>> >>> 2. What's the state of the System? are the clients receiving the >>> messages. The queue seems to be holding a very large number of >>> objects. >>> Essentially what I want to know is, if the clients are receiving the >>> messages or the Server is holding them up. >>> >>> Will spend more time with the issue and see what I can figure out. >>> >>> thanks >>> ashish >>> >>> On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer >>> wrote: >>> Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish: > > Can you provide the heapdump for this OOM? > > thanks > ashish > > On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: > >> >> On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer >> wrote: >> >>> >>> Hi Ashish, >>> >>> I think we tracked down the source of the problem a bit more.. The OOM >>> seems to be related to IMAP. Our IMAP server component is using the >>> StreamIoHandler (its the only one of our components who use this >>> handler). So I suspect there is the problem. >>> >>> So there are two possible problems: >>> 1) Bug in StreamIoHandler >>> 2) Wrong usage of StreamIoHandler. Our implementations is here: >>> >>> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >>> >>> Thx, >>> Norman >>> >> >> Sorry, haven't been able to look at this so far :( >> earliest I can give it a shot will be on Sunday. >> >> thanks >> ashish >> > > >>> >>> >>> >> >> >> - >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> Fo
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman 2010/4/12 Ashish : > On Sun, Apr 11, 2010 at 8:57 PM, Norman Maurer > wrote: >> Hi Ashish, >> >> after thinkin more about the whole stuff I wonder if the exception >> handling in SMTP could sometimes be an endless loop. In the handler we >> write a message back to the client on exception. But what happens if >> this throw a new exception ? Could it so loop forever ? > > I am not sure, but it could be possible. > >> >> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014&view=markup >> >> Maybe we should just close the connection.. > > hmm.. its brute force. I am not convinced that we should do this, but > not sure as I don't have a complete picture as such. > If the exception is coz of packet decoding, ignore the packet and move > on.. Its also driven by the protocol as such. You may need > to respond back to the client with some error code etc. > > Eric: Please mark a copy to mina dev ML as well, else I don't get your mails > :) > > The analysis is a session is in hanging state, whose queue is building > up. Heap dump are not going to help any more. > Can you trace down what the hack is going on with that session. I > can't get much out of heap dump and unfortunately not able to run it > on laptop. > jconsole simply refuse to detect the James process :( > > So the key lies in analyzing > 1. Why is this session hanging, its simply not writing the data back > to the client. There is a while before the session is closed and > session.isConnected() returns false. > > Eric: can you upload the logs as well, if possible :) > > Anyone else has some idea's here? > > thanks > ashish > >> >> Bye >> Norman >> >> 2010/4/9 Eric Charles : >>> Hi Ashish, >>> >>> I am the user who has many OOM with the trunk and feed Norman with my issues >>> :) >>> >>> 1. >>> Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is >>> the release we normally use in james and give the issue after a certain >>> period of time. I mean, the server can stay working correctly 2 days or >>> crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due >>> to a peak in memory usage. T >>> To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, >>> we see a direct crash. I've uploaded such a dump on >>> http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force >>> refresh, I setup the web server in a hurry). You will see there recursive >>> ConcurrentLinkedQueue. >>> >>> 2. >>> As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works >>> while the line is flat (mail are received, spooled, delivered,...). On a few >>> seconds, it peaks and of course, nothing works anymore. So, the messages >>> don't remain in the spool. I made tests with quite huge messages, and they >>> are delivered very fast as soon as they arrive in James. >>> >>> Don't hesitate to ask more questions or propose additional tests. >>> >>> Many tks in advance, >>> >>> Eric >>> >>> >>> On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer wrote: > > Maybe Eric can do, cause he is the one who see it very freqently.. > > So Eric...;) ? > > Thx, > Norman > > > 2010/4/8 Ashish: > >> >> Can you provide the heapdump for this OOM? >> >> thanks >> ashish >> >> On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: >> >>> >>> On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer >>> wrote: >>> Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserv
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, after thinkin more about the whole stuff I wonder if the exception handling in SMTP could sometimes be an endless loop. In the handler we write a message back to the client on exception. But what happens if this throw a new exception ? Could it so loop forever ? http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014&view=markup Maybe we should just close the connection.. Bye Norman 2010/4/9 Eric Charles : > Hi Ashish, > > I am the user who has many OOM with the trunk and feed Norman with my issues > :) > > 1. > Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is > the release we normally use in james and give the issue after a certain > period of time. I mean, the server can stay working correctly 2 days or > crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due > to a peak in memory usage. T > To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, > we see a direct crash. I've uploaded such a dump on > http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force > refresh, I setup the web server in a hurry). You will see there recursive > ConcurrentLinkedQueue. > > 2. > As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works > while the line is flat (mail are received, spooled, delivered,...). On a few > seconds, it peaks and of course, nothing works anymore. So, the messages > don't remain in the spool. I made tests with quite huge messages, and they > are delivered very fast as soon as they arrive in James. > > Don't hesitate to ask more questions or propose additional tests. > > Many tks in advance, > > Eric > > > On 04/09/2010 12:40 PM, Ashish wrote: >> >> Norman, >> >> Couple of more queries >> >> 1. The heap dump uses circularqueue class, so seem to be taken for an >> earlier trunk snapshot. Is my take correct? >> >> 2. What's the state of the System? are the clients receiving the >> messages. The queue seems to be holding a very large number of >> objects. >> Essentially what I want to know is, if the clients are receiving the >> messages or the Server is holding them up. >> >> Will spend more time with the issue and see what I can figure out. >> >> thanks >> ashish >> >> On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer >> wrote: >> >>> >>> Maybe Eric can do, cause he is the one who see it very freqently.. >>> >>> So Eric...;) ? >>> >>> Thx, >>> Norman >>> >>> >>> 2010/4/8 Ashish: >>> Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: > > On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer > wrote: > >> >> Hi Ashish, >> >> I think we tracked down the source of the problem a bit more.. The OOM >> seems to be related to IMAP. Our IMAP server component is using the >> StreamIoHandler (its the only one of our components who use this >> handler). So I suspect there is the problem. >> >> So there are two possible problems: >> 1) Bug in StreamIoHandler >> 2) Wrong usage of StreamIoHandler. Our implementations is here: >> >> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >> >> Thx, >> Norman >> > > Sorry, haven't been able to look at this so far :( > earliest I can give it a shot will be on Sunday. > > thanks > ashish > >>> >>> >> >> >> > > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Lets dig this further and finish this off.. :) Will try to generate this OOM locally... will keep you all posted On Fri, Apr 9, 2010 at 9:32 PM, Norman Maurer wrote: > Hi Ashish, > yeah ofter some more investigation it seems that it's not the > StreamIoHandler which cause the problem. It must be something else... > > Bye > Norman > > > > 2010/4/9 Ashish : >> Did some more investigation. Here is one of the strange observations >> >> Most of the hanging DefaultWriteRequest have this message "451 Unable >> to process smtp request.." >> Can't analyze all of them, but saw content of close to 100 objects. >> All of them had the same content >> >> This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler >> Line:94 >> >> if (session.isConnected()) { >> session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, "Unable to >> process smtp request")); >> } >> >> Also, from the heapdump, all the WriteRequests are for a session >> (id=13609), and they are hanging there :( >> so seems like we have a hanging session. >> >> Any thoughts? >> >> Will try to reproduce this problem at my end. Some logs might be of help. >> >> thanks >> ashish >> >> >> >> >> On Fri, Apr 9, 2010 at 4:10 PM, Ashish wrote: >>> Norman, >>> >>> Couple of more queries >>> >>> 1. The heap dump uses circularqueue class, so seem to be taken for an >>> earlier trunk snapshot. Is my take correct? >>> >>> 2. What's the state of the System? are the clients receiving the >>> messages. The queue seems to be holding a very large number of >>> objects. >>> Essentially what I want to know is, if the clients are receiving the >>> messages or the Server is holding them up. >>> >>> Will spend more time with the issue and see what I can figure out. >>> >>> thanks >>> ashish >>> >>> On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer >>> wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish : > Can you provide the heapdump for this OOM? > > thanks > ashish > > On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: >> On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer >> wrote: >>> Hi Ashish, >>> >>> I think we tracked down the source of the problem a bit more.. The OOM >>> seems to be related to IMAP. Our IMAP server component is using the >>> StreamIoHandler (its the only one of our components who use this >>> handler). So I suspect there is the problem. >>> >>> So there are two possible problems: >>> 1) Bug in StreamIoHandler >>> 2) Wrong usage of StreamIoHandler. Our implementations is here: >>> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >>> >>> Thx, >>> Norman >> >> Sorry, haven't been able to look at this so far :( >> earliest I can give it a shot will be on Sunday. >> >> thanks >> ashish > >>> >>> >>> >>> -- >>> thanks >>> ashish >>> >>> Blog: http://www.ashishpaliwal.com/blog >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, yeah ofter some more investigation it seems that it's not the StreamIoHandler which cause the problem. It must be something else... Bye Norman 2010/4/9 Ashish : > Did some more investigation. Here is one of the strange observations > > Most of the hanging DefaultWriteRequest have this message "451 Unable > to process smtp request.." > Can't analyze all of them, but saw content of close to 100 objects. > All of them had the same content > > This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler > Line:94 > > if (session.isConnected()) { >session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, "Unable to > process smtp request")); > } > > Also, from the heapdump, all the WriteRequests are for a session > (id=13609), and they are hanging there :( > so seems like we have a hanging session. > > Any thoughts? > > Will try to reproduce this problem at my end. Some logs might be of help. > > thanks > ashish > > > > > On Fri, Apr 9, 2010 at 4:10 PM, Ashish wrote: >> Norman, >> >> Couple of more queries >> >> 1. The heap dump uses circularqueue class, so seem to be taken for an >> earlier trunk snapshot. Is my take correct? >> >> 2. What's the state of the System? are the clients receiving the >> messages. The queue seems to be holding a very large number of >> objects. >> Essentially what I want to know is, if the clients are receiving the >> messages or the Server is holding them up. >> >> Will spend more time with the issue and see what I can figure out. >> >> thanks >> ashish >> >> On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer >> wrote: >>> Maybe Eric can do, cause he is the one who see it very freqently.. >>> >>> So Eric...;) ? >>> >>> Thx, >>> Norman >>> >>> >>> 2010/4/8 Ashish : Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: > On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer > wrote: >> Hi Ashish, >> >> I think we tracked down the source of the problem a bit more.. The OOM >> seems to be related to IMAP. Our IMAP server component is using the >> StreamIoHandler (its the only one of our components who use this >> handler). So I suspect there is the problem. >> >> So there are two possible problems: >> 1) Bug in StreamIoHandler >> 2) Wrong usage of StreamIoHandler. Our implementations is here: >> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >> >> Thx, >> Norman > > Sorry, haven't been able to look at this so far :( > earliest I can give it a shot will be on Sunday. > > thanks > ashish >>> >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Did some more investigation. Here is one of the strange observations Most of the hanging DefaultWriteRequest have this message "451 Unable to process smtp request.." Can't analyze all of them, but saw content of close to 100 objects. All of them had the same content This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler Line:94 if (session.isConnected()) { session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, "Unable to process smtp request")); } Also, from the heapdump, all the WriteRequests are for a session (id=13609), and they are hanging there :( so seems like we have a hanging session. Any thoughts? Will try to reproduce this problem at my end. Some logs might be of help. thanks ashish On Fri, Apr 9, 2010 at 4:10 PM, Ashish wrote: > Norman, > > Couple of more queries > > 1. The heap dump uses circularqueue class, so seem to be taken for an > earlier trunk snapshot. Is my take correct? > > 2. What's the state of the System? are the clients receiving the > messages. The queue seems to be holding a very large number of > objects. > Essentially what I want to know is, if the clients are receiving the > messages or the Server is holding them up. > > Will spend more time with the issue and see what I can figure out. > > thanks > ashish > > On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer > wrote: >> Maybe Eric can do, cause he is the one who see it very freqently.. >> >> So Eric...;) ? >> >> Thx, >> Norman >> >> >> 2010/4/8 Ashish : >>> Can you provide the heapdump for this OOM? >>> >>> thanks >>> ashish >>> >>> On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer wrote: > Hi Ashish, > > I think we tracked down the source of the problem a bit more.. The OOM > seems to be related to IMAP. Our IMAP server component is using the > StreamIoHandler (its the only one of our components who use this > handler). So I suspect there is the problem. > > So there are two possible problems: > 1) Bug in StreamIoHandler > 2) Wrong usage of StreamIoHandler. Our implementations is here: > http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup > > Thx, > Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish >>> >> > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer wrote: > Maybe Eric can do, cause he is the one who see it very freqently.. > > So Eric...;) ? > > Thx, > Norman > > > 2010/4/8 Ashish : >> Can you provide the heapdump for this OOM? >> >> thanks >> ashish >> >> On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: >>> On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer >>> wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman >>> >>> Sorry, haven't been able to look at this so far :( >>> earliest I can give it a shot will be on Sunday. >>> >>> thanks >>> ashish >> > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, You can find some information to download on http://apache.u-mangate.com/james/oom/index.html Tks, Eric On 04/08/2010 10:26 AM, Ashish wrote: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: > On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer > wrote: >> Hi Ashish, >> >> I think we tracked down the source of the problem a bit more.. The OOM >> seems to be related to IMAP. Our IMAP server component is using the >> StreamIoHandler (its the only one of our components who use this >> handler). So I suspect there is the problem. >> >> So there are two possible problems: >> 1) Bug in StreamIoHandler >> 2) Wrong usage of StreamIoHandler. Our implementations is here: >> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >> >> Thx, >> Norman > > Sorry, haven't been able to look at this so far :( > earliest I can give it a shot will be on Sunday. > > thanks > ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish : > Can you provide the heapdump for this OOM? > > thanks > ashish > > On Thu, Apr 8, 2010 at 1:40 PM, Ashish wrote: >> On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer >> wrote: >>> Hi Ashish, >>> >>> I think we tracked down the source of the problem a bit more.. The OOM >>> seems to be related to IMAP. Our IMAP server component is using the >>> StreamIoHandler (its the only one of our components who use this >>> handler). So I suspect there is the problem. >>> >>> So there are two possible problems: >>> 1) Bug in StreamIoHandler >>> 2) Wrong usage of StreamIoHandler. Our implementations is here: >>> http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup >>> >>> Thx, >>> Norman >> >> Sorry, haven't been able to look at this so far :( >> earliest I can give it a shot will be on Sunday. >> >> thanks >> ashish > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer wrote: > Hi Ashish, > > I think we tracked down the source of the problem a bit more.. The OOM > seems to be related to IMAP. Our IMAP server component is using the > StreamIoHandler (its the only one of our components who use this > handler). So I suspect there is the problem. > > So there are two possible problems: > 1) Bug in StreamIoHandler > 2) Wrong usage of StreamIoHandler. Our implementations is here: > http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup > > Thx, > Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman 2010/4/5 Eric Charles : > Yes, > > One instance of > org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue > contains : > - for 2.0.0-RC1 : one instance of org.apache.mina.util.CircularQueue with an > array of many > org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest > - for 2.0.0-RC2-SNAPSHOT : many recursive instances of > java.util.concurrent.ConcurrentLinkedQueue > > Tks, > Eric > > > On 04/05/2010 10:51 AM, Norman Maurer wrote: >> >> I think thats only because mina-trunk is not using CircularQueue >> anymore. It now seems to use ConcurrentLinkedQueue. >> >> Bye, >> Norman >> >> 2010/4/5 Eric Charles: >> >>> >>> Hi Ashish, >>> >>> Please also note that we have 2 different exceptions: >>> - http://people.apache.org/~norman/oom/Screenshot.png that occurs with >>> mina >>> 2.0.0-RC1 >>> - >>> >>> http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png >>> that occurs with 2.0.0-RC2-SNAPSHOT >>> >>> It happens on jdk 1.6 (and also 1.7) >>> >>> Tks, >>> >>> Eric >>> >>> >>> On 04/05/2010 10:01 AM, Ashish wrote: >>> Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer wrote: > > Hi Ashish, > > thx for your reply. The problem is sometimes it takes sometime to get > the OOM, other times it happens within one minute. One of our users > reported to me that it happened within 1 minute (10 seconds and 30 > seconds), after he switched to MINA trunk. So it seems it is faster > reproducable with MINA trunk. I uploaded the screenshot of Eclipse > Memory Analyzer to : > > > > http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png > > This was the one we got with MINA trunk. So you it again shows all the > memory allocated by MINA. > > To reproduce the OOM you must perform the following steps: > > 1) svn checkout http://svn.apache.org/repos/asf/server/trunk > james-trunk > 2) cd james-trunk > 2) mvn clean package > 3) tar xfvz > spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz > 4) cd james-string-deploymnet-3.0-M1/bin > 5) sudo ./run.sh > 6) telnet localhost 4555 (user root, pass root) > 7) adduser test test > 8) quit > > Now bomb the server with emails. You can use for example smtp-source > for it (wich is included in postfix) > 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test > localhost:25 > > This will bomb the servers with 20 emails with 20 concurrent > sessions. > > Hope this helps, > Bye, > Norman > > 2010/4/5 Ashish: > > >> >> On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer >> wrote: >> >> >>> >>> Hi all, >>> >>> we at JAMES use MINA since a while now for our socket stuff. After >>> doing some stress tests we encountered some OOM Exceptions. The same >>> was seen by one of our users which use JAMES trunk. After debugging >>> stuff it seems like the cause of the OOM is MINA. We took some heap >>> which shows that MINA is takin the most memory. The class which shows >>> the memory usage was: >>> >>> org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest >>> >>> I attach the screnshots which shows the problem. Any idea ? We are >>> using MINA-2.0.0-RC1. >>> >>> Maybe I'm wrong and the whole OOM is not related to MINA at all.. >>> >>> Thanks, >>> Norman >>> >>> Ps: Please keep server-dev in the cc >>> >>> >> >> Norman, >> >> Is there a way to reproduce this? I just want to try it out myself and >> debug. >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> >> >> > > >>> >>> - >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >>> >>> >> >> - >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>
Re: JAMES OOM, maybe the cause of MINA ?
On Mon, Apr 5, 2010 at 2:21 PM, Norman Maurer wrote: > I think thats only because mina-trunk is not using CircularQueue > anymore. It now seems to use ConcurrentLinkedQueue. > > Bye, > Norman I remember, Emm removed all references to CircularQueue, but this was to make it thread safe :) http://mina.markmail.org/message/xirxdepng3j65w4b?q=CircularQueue#query:CircularQueue+page:1+mid:xirxdepng3j65w4b+state:results Hmm, not sure why it would hold the references. Thanks for Heads up.. > > 2010/4/5 Eric Charles : >> Hi Ashish, >> >> Please also note that we have 2 different exceptions: >> - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina >> 2.0.0-RC1 >> - >> http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png >> that occurs with 2.0.0-RC2-SNAPSHOT >> >> It happens on jdk 1.6 (and also 1.7) >> >> Tks, >> >> Eric >> >> >> On 04/05/2010 10:01 AM, Ashish wrote: >>> >>> Thanks Norman, >>> >>> Will try to see this as soon as I can get some time from paid job :) >>> >>> cheers >>> ashish >>> >>> On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer >>> wrote: >>> Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish: > > On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer > wrote: > >> >> Hi all, >> >> we at JAMES use MINA since a while now for our socket stuff. After >> doing some stress tests we encountered some OOM Exceptions. The same >> was seen by one of our users which use JAMES trunk. After debugging >> stuff it seems like the cause of the OOM is MINA. We took some heap >> which shows that MINA is takin the most memory. The class which shows >> the memory usage was: >> >> org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest >> >> I attach the screnshots which shows the problem. Any idea ? We are >> using MINA-2.0.0-RC1. >> >> Maybe I'm wrong and the whole OOM is not related to MINA at all.. >> >> Thanks, >> Norman >> >> Ps: Please keep server-dev in the cc >> > > Norman, > > Is there a way to reproduce this? I just want to try it out myself and > debug. > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > > >>> >>> >>> >> >> >> - >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Yes, One instance of org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue contains : - for 2.0.0-RC1 : one instance of org.apache.mina.util.CircularQueue with an array of many org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest - for 2.0.0-RC2-SNAPSHOT : many recursive instances of java.util.concurrent.ConcurrentLinkedQueue Tks, Eric On 04/05/2010 10:51 AM, Norman Maurer wrote: I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charles: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charles : > Hi Ashish, > > Please also note that we have 2 different exceptions: > - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina > 2.0.0-RC1 > - > http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png > that occurs with 2.0.0-RC2-SNAPSHOT > > It happens on jdk 1.6 (and also 1.7) > > Tks, > > Eric > > > On 04/05/2010 10:01 AM, Ashish wrote: >> >> Thanks Norman, >> >> Will try to see this as soon as I can get some time from paid job :) >> >> cheers >> ashish >> >> On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer >> wrote: >> >>> >>> Hi Ashish, >>> >>> thx for your reply. The problem is sometimes it takes sometime to get >>> the OOM, other times it happens within one minute. One of our users >>> reported to me that it happened within 1 minute (10 seconds and 30 >>> seconds), after he switched to MINA trunk. So it seems it is faster >>> reproducable with MINA trunk. I uploaded the screenshot of Eclipse >>> Memory Analyzer to : >>> >>> >>> http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png >>> >>> This was the one we got with MINA trunk. So you it again shows all the >>> memory allocated by MINA. >>> >>> To reproduce the OOM you must perform the following steps: >>> >>> 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk >>> 2) cd james-trunk >>> 2) mvn clean package >>> 3) tar xfvz >>> spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz >>> 4) cd james-string-deploymnet-3.0-M1/bin >>> 5) sudo ./run.sh >>> 6) telnet localhost 4555 (user root, pass root) >>> 7) adduser test test >>> 8) quit >>> >>> Now bomb the server with emails. You can use for example smtp-source >>> for it (wich is included in postfix) >>> 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test >>> localhost:25 >>> >>> This will bomb the servers with 20 emails with 20 concurrent >>> sessions. >>> >>> Hope this helps, >>> Bye, >>> Norman >>> >>> 2010/4/5 Ashish: >>> On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: > > Hi all, > > we at JAMES use MINA since a while now for our socket stuff. After > doing some stress tests we encountered some OOM Exceptions. The same > was seen by one of our users which use JAMES trunk. After debugging > stuff it seems like the cause of the OOM is MINA. We took some heap > which shows that MINA is takin the most memory. The class which shows > the memory usage was: > > org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest > > I attach the screnshots which shows the problem. Any idea ? We are > using MINA-2.0.0-RC1. > > Maybe I'm wrong and the whole OOM is not related to MINA at all.. > > Thanks, > Norman > > Ps: Please keep server-dev in the cc > Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >>> >> >> >> > > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer wrote: > Hi Ashish, > > thx for your reply. The problem is sometimes it takes sometime to get > the OOM, other times it happens within one minute. One of our users > reported to me that it happened within 1 minute (10 seconds and 30 > seconds), after he switched to MINA trunk. So it seems it is faster > reproducable with MINA trunk. I uploaded the screenshot of Eclipse > Memory Analyzer to : > > http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png > > This was the one we got with MINA trunk. So you it again shows all the > memory allocated by MINA. > > To reproduce the OOM you must perform the following steps: > > 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk > 2) cd james-trunk > 2) mvn clean package > 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz > 4) cd james-string-deploymnet-3.0-M1/bin > 5) sudo ./run.sh > 6) telnet localhost 4555 (user root, pass root) > 7) adduser test test > 8) quit > > Now bomb the server with emails. You can use for example smtp-source > for it (wich is included in postfix) > 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test > localhost:25 > > This will bomb the servers with 20 emails with 20 concurrent sessions. > > Hope this helps, > Bye, > Norman > > 2010/4/5 Ashish : >> On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: >>> Hi all, >>> >>> we at JAMES use MINA since a while now for our socket stuff. After >>> doing some stress tests we encountered some OOM Exceptions. The same >>> was seen by one of our users which use JAMES trunk. After debugging >>> stuff it seems like the cause of the OOM is MINA. We took some heap >>> which shows that MINA is takin the most memory. The class which shows >>> the memory usage was: >>> >>> org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest >>> >>> I attach the screnshots which shows the problem. Any idea ? We are >>> using MINA-2.0.0-RC1. >>> >>> Maybe I'm wrong and the whole OOM is not related to MINA at all.. >>> >>> Thanks, >>> Norman >>> >>> Ps: Please keep server-dev in the cc >> >> Norman, >> >> Is there a way to reproduce this? I just want to try it out myself and debug. >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish : > On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: >> Hi all, >> >> we at JAMES use MINA since a while now for our socket stuff. After >> doing some stress tests we encountered some OOM Exceptions. The same >> was seen by one of our users which use JAMES trunk. After debugging >> stuff it seems like the cause of the OOM is MINA. We took some heap >> which shows that MINA is takin the most memory. The class which shows >> the memory usage was: >> >> org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest >> >> I attach the screnshots which shows the problem. Any idea ? We are >> using MINA-2.0.0-RC1. >> >> Maybe I'm wrong and the whole OOM is not related to MINA at all.. >> >> Thanks, >> Norman >> >> Ps: Please keep server-dev in the cc > > Norman, > > Is there a way to reproduce this? I just want to try it out myself and debug. > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer wrote: > Hi all, > > we at JAMES use MINA since a while now for our socket stuff. After > doing some stress tests we encountered some OOM Exceptions. The same > was seen by one of our users which use JAMES trunk. After debugging > stuff it seems like the cause of the OOM is MINA. We took some heap > which shows that MINA is takin the most memory. The class which shows > the memory usage was: > > org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest > > I attach the screnshots which shows the problem. Any idea ? We are > using MINA-2.0.0-RC1. > > Maybe I'm wrong and the whole OOM is not related to MINA at all.. > > Thanks, > Norman > > Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
I should think of the attachment stripping on the ml.. You can find the screenshots here: http://people.apache.org/~norman/oom/ Bye, Norman 2010/4/3 Norman Maurer : > Hi all, > > we at JAMES use MINA since a while now for our socket stuff. After > doing some stress tests we encountered some OOM Exceptions. The same > was seen by one of our users which use JAMES trunk. After debugging > stuff it seems like the cause of the OOM is MINA. We took some heap > which shows that MINA is takin the most memory. The class which shows > the memory usage was: > > org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest > > I attach the screnshots which shows the problem. Any idea ? We are > using MINA-2.0.0-RC1. > > Maybe I'm wrong and the whole OOM is not related to MINA at all.. > > Thanks, > Norman > > Ps: Please keep server-dev in the cc > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org