[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734021#action_12734021
]
Victor N commented on DIRMINA-709:
--
Edouard, your patch works well, but ClosedChannelException still occurs. It is
not critical anyway.
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>Assignee: Edouard De Oliveira
> Fix For: 2.0.0-M7
>
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715810#action_12715810
]
Victor N commented on DIRMINA-709:
--
Our production servers are working with this fix during ~1 month, everything
seems to work.
The only side-effect is mentioned in my comment above - sometimes, rarely, I
can see this warning in log files, but everything is working.
Forgot to say that I disabled MdcInjectionFilterTest when building mina with
the patch because this test failed.
Are there any plans to review this fix and include it into the next build -
maybe M7 ?
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705998#action_12705998
]
Victor N commented on DIRMINA-709:
--
The only side-effect I see is that sometimes after remove() method there is an
exception:
04/05/2009 14:25:18| WARN |
org.apache.mina.util.DefaultExceptionMonitor.exceptionCaught(): Unexpected
exception.
java.nio.channels.ClosedChannelException
at
java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
at
org.apache.mina.transport.socket.nio.NioProcessor.init(NioProcessor.java:100)
at
org.apache.mina.transport.socket.nio.NioProcessor.init(NioProcessor.java:42)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.addNow(AbstractPollingIoProcessor.java:417)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.add(AbstractPollingIoProcessor.java:403)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$200(AbstractPollingIoProcessor.java:59)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:878)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Maybe this is somehow related to my patch, I do not know. Maybe we could think
at a better patch.
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705997#action_12705997
]
Victor N commented on DIRMINA-709:
--
My patch is the following - I have changed AbstractPollingIoProcessor.remove()
method this way:
switch (state) {
case OPEN:
case PREPARING:
if (removeNow(session)) {
removedSessions ++;
}
break;
case CLOSED:
// Skip if channel is already closed
break;
I am not sure whether the problem was for every PREPARING session or not, but
the patch seems to work.
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705996#action_12705996
]
Victor N commented on DIRMINA-709:
--
Seems that my patch is working - I can see that several pending sessions were
removed during last 3 days on our servers.
I hope that it does not have impact on some other functions of mina :)
Anyway, it would be great to see comments from mina creators!
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DIRMINA-709) PENDING session is removed and added endlessly -- garbage allocation and high CPU usage problem
[
https://issues.apache.org/jira/browse/DIRMINA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705995#action_12705995
]
Victor N commented on DIRMINA-709:
--
If I do a simple workaround - call removeNow() for such sessions - can
something go wrong this way?
> PENDING session is removed and added endlessly -- garbage allocation and
> high CPU usage problem
>
>
> Key: DIRMINA-709
> URL: https://issues.apache.org/jira/browse/DIRMINA-709
> Project: MINA
> Issue Type: Bug
> Components: Core
>Affects Versions: 2.0.0-M4
> Environment: Debian Linux, kernel 2.6.24
>Reporter: Victor N
>
> (This problem was discussed in mail lists, I will copy it here).
> Seems I have found a bug with IoSession - I can see that a "PREPARING"
> session is not removed correctly from queue.
> When some session is in PREPARING state, it is removed from
> "removingSessions" queue but right after that it is added to this queue
> again! So this session is added to the queue and removed from it until
> forever.
> As a result, this give us significant garbage allocation, so CPU spends most
> of time in garbage collector (I can see this is JConsole).
> I see comments there in AbstractPollingIoProcessor class:
> private int remove() {
> ...
> case PREPARING:
> // Retry later if session is not yet fully initialized.
> // (In case that Session.close() is called before
> addSession() is processed)
> scheduleRemove(session);
> return removedSessions;
> ...
> }
> I have added logging to this code, and I can see that the SAME session is
> removed and added again and again.
> Can somebody explain this logic please? Why don't we remove the PENDING
> session? Or maybe is there a workaround for this.
> Sorry, I can not provide a test for this issue, but it is reproduced almost
> every day at out production servers under some load. Maybe, you can reproduce
> it by adding a delay in addSession() and then closing the session during this
> delay.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
