RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
Any takers to tackle this issue?
-Original Message-
From: Serge Perepel [mailto:se...@american-data.com]
Sent: Friday, January 26, 2018 2:33 PM
To: Tomcat Users List
Subject: RE: Suspected memory leak of
org.apache.coyote.AbstractProtocol$ConnectionHandler object while using
Websocket
-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
>This is likely to get looked at faster if you provide code that other people
>can run that will reproduce the issue you are seeing rather than expecting
>someone else to construct the test case for you.
Here is how you can reproduce it:
Server websocket:
package ad.ecs.websocket;
import java.io.IOException;
import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
@ServerEndpoint(value = "/asyncMsg")
public class TestWebsocket {
@OnOpen
public void open(Session session) throws IOException{
session.getBasicRemote().sendText("Connection Established");
}
@OnMessage
public String login(String sessionID, Session session) {
//you can do this instead of exception and it will still leak memory
//try {
//session.close(new CloseReason(CloseCodes.GOING_AWAY, "Clearing
session"));
//} catch (IOException e) {
//e.printStackTrace();
//}
throw new IllegalArgumentException("Testing Leak");
}
@OnClose
public void close(Session session, CloseReason reason) {
}
@OnError
public void error(Session session, Throwable error) {
error.printStackTrace();
}
}
Here is the front end html:
LEAK
Here is js file:
var webSocket = null;
function test() {
console.log("test button clicked");
openSocket();
}
function openSocket() {
console.log("open webSocket");
webSocket = new WebSocket("ws://" + window.location.host +
"/memoryleak/asyncMsg");
webSocket.onopen = function(event){
console.log("webSocket open");
webSocket.send('test');
};
webSocket.onmessage = function(event){
console.log("webSocket onMessage", event);
};
webSocket.onclose = function(event){
webSocket = null;
console.log("webSocket connection closed", event);
openSocket();
};
}
function closeSocket(){
console.log("close webSocket");
if (webSocket)
webSocket.close();
}
You need to run it on multiple clients if you want it leak fast. We did it with
8 clients and it leaked 3GB with in like 10 min.
Thank you
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]
Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
se...@american-data.com<mailto:se...@american-data.com> | www.american-data.com
[Follow us on
Facebook]<https://www.facebook.com/pages/American-Data/10044518233>
[Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>
[Follow us on Twitter] <https://twitter.com/AmericanDataECS> [Follow us on
Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>
Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are
not the intended recipient, you should not use, disclose or take any action
based on the message. If you have received this transmission in error, please
immediately contact the sender by return e-mail and delete this e-mail, and any
attachments, from any computer.
[ECS 10 Conversion]<http://www.american-data.com/viewer>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
>This is likely to get looked at faster if you provide code that other people
>can run that will reproduce the issue you are seeing rather than expecting
>someone else to construct the test case for you.
Here is how you can reproduce it:
Server websocket:
package ad.ecs.websocket;
import java.io.IOException;
import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
@ServerEndpoint(value = "/asyncMsg")
public class TestWebsocket {
@OnOpen
public void open(Session session) throws IOException{
session.getBasicRemote().sendText("Connection Established");
}
@OnMessage
public String login(String sessionID, Session session) {
//you can do this instead of exception and it will still leak memory
//try {
//session.close(new CloseReason(CloseCodes.GOING_AWAY, "Clearing
session"));
//} catch (IOException e) {
//e.printStackTrace();
//}
throw new IllegalArgumentException("Testing Leak");
}
@OnClose
public void close(Session session, CloseReason reason) {
}
@OnError
public void error(Session session, Throwable error) {
error.printStackTrace();
}
}
Here is the front end html:
LEAK
Here is js file:
var webSocket = null;
function test() {
console.log("test button clicked");
openSocket();
}
function openSocket() {
console.log("open webSocket");
webSocket = new WebSocket("ws://" + window.location.host +
"/memoryleak/asyncMsg");
webSocket.onopen = function(event){
console.log("webSocket open");
webSocket.send('test');
};
webSocket.onmessage = function(event){
console.log("webSocket onMessage", event);
};
webSocket.onclose = function(event){
webSocket = null;
console.log("webSocket connection closed", event);
openSocket();
};
}
function closeSocket(){
console.log("close webSocket");
if (webSocket)
webSocket.close();
}
You need to run it on multiple clients if you want it leak fast. We did it with
8 clients and it leaked 3GB with in like 10 min.
Thank you
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]
Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
se...@american-data.com | www.american-data.com
[Follow us on
Facebook]
[Follow us on LinkedIn]
[Follow us on Twitter] [Follow us on
Twitter]
Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are
not
the intended recipient, you should not use, disclose or take any action based on
the message. If you have received this transmission in error, please immediately
contact the sender by return e-mail and delete this e-mail,
and any attachments, from any computer.
[ECS 10 Conversion]
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
On 26/01/18 15:11, Serge Perepel wrote:
> When we run our app for 2-3 hours we experience this leak related to our web
> sockets connections, here is the Memory Analyzer Suspect:
>
> One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded
> by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%)
> bytes. The memory is accumulated in one instance of
> "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by " loader>".
A break down of that ~2Gb would be helpful. How many entries in the Map?
What is the distribution of sizes?
> Keywords
> org.apache.coyote.AbstractProtocol$ConnectionHandler
> java.util.concurrent.ConcurrentHashMap$Node[]
> java.net.URLClassLoader @ 0x720029098
>
>
> Also these bugs might be related to this one:
>
>
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57546
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57750
>
>
>
> The 57546 bug looks very similar to what we are experiencing. We tested on
> Linux and so far we do not see the same behavior. Also this is our web socket
> code:
>
>
>
> package ad.ecs.async.websocket;
>
>
>
> import java.io.IOException;
>
> import java.util.Arrays;
>
>
>
> import javax.websocket.CloseReason;
>
> import javax.websocket.OnClose;
>
> import javax.websocket.OnError;
>
> import javax.websocket.OnMessage;
>
> import javax.websocket.OnOpen;
>
> import javax.websocket.Session;
>
> import javax.websocket.server.ServerEndpoint;
>
>
>
> import ad.common.Global;
>
> import ad.ecs.async.AsyncEngine;
>
> import ad.ecs.async.AsyncResponse;
>
> import ad.ecs.async.AsyncType;
>
> import ad.ecs.db.DatabaseEngine;
>
> import ad.ecs.db.paradox.User;
>
> import ad.ecs.security.engine.SecurityEngine;
>
>
>
> /**
>
> * @author Serge Perepel
>
> * @since Aug 23, 2017 12:23:32 PM
>
> */
>
> @ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)
>
> public class ECSAsync {
>
>
>
> @OnOpen
>
> public void open(Session session) throws IOException{
>
> session.getBasicRemote().sendText("Connection Established");
>
> }
>
>
>
> @OnMessage
>
> public String login(String sessionID, Session session) {
>
> AsyncEngine.INSTANCE.wsConnect(session, sessionID);
>
> org.hibernate.Session dbSession =
> DatabaseEngine.getSessionFactory().openSession();
>
> try {
>
>int userID =
> SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);
>
>User user = (User) dbSession.get(User.class, userID);
>
>if (user != null) {
>
>if (user.getNextLogin() == 1) {
>
>AsyncResponse response = new AsyncResponse();
>
>response.setType(AsyncType.None);
>
>response.setData(Arrays.asList("PASSWORD"));
>
>response.setObjData("PASSWORD");
>
>
> AsyncEngine.INSTANCE.addTransientResult(sessionID, response);
>
>}
>
>}
>
> } finally {
>
>dbSession.close();
>
> }
>
> return "ok";
>
> }
>
>
>
> @OnClose
>
> public void close(Session session, CloseReason reason) {
>
> AsyncEngine.INSTANCE.wsDisconnect(session);
>
> }
>
>
>
> @OnError
>
> public void error(Session session, Throwable error) {
>
> Global.INSTANCE.getLogHelper().exception(error);
>
> //session.close(new CloseReason(closeCode, reasonPhrase));
>
> }
>
> }
>
>
>
> Front end opens connection and sends invalid sessionID which causes
>
> the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw
> exception and after that disconnect happens on the front end. At this point
> front end opens new connection and process goes into the loop. I'm assuming
> that the connection handler suppose to get freed after a disconnect happen.
> But it seems to accumulate. You can try this code and just replace code in
> the login method to always throw the exception. On the front end onDisconnect
> try to open new connection and send a random message to the web socket.
This is likely to get looked at faster if you provide code that other
people can run that will reproduce the issue you are seeing rather than
expecting someone else to construct the test case for you.
Mark
>
>
>
> Thank you
>
> Serge
>
> [http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]
>
> Serge Perepel
> Software Developer | American Data
> p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
> se...@american-data.com |
> www.american-data.com
>
>
> [Follow us on
> Facebook]
> [Follow us on LinkedIn]
> [Follow us on Twitter
RE: Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
Forgot to mention that we are running Tomcat 8.5 on Windows 2012 Server
-Original Message-
From: Serge Perepel [mailto:se...@american-data.com]
Sent: Friday, January 26, 2018 9:12 AM
To: users@tomcat.apache.org
Subject: Suspected memory leak of
org.apache.coyote.AbstractProtocol$ConnectionHandler object while using
Websocket
When we run our app for 2-3 hours we experience this leak related to our web
sockets connections, here is the Memory Analyzer Suspect:
One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded
by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%)
bytes. The memory is accumulated in one instance of
"java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "".
Keywords
org.apache.coyote.AbstractProtocol$ConnectionHandler
java.util.concurrent.ConcurrentHashMap$Node[]
java.net.URLClassLoader @ 0x720029098
Also these bugs might be related to this one:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57546
https://bz.apache.org/bugzilla/show_bug.cgi?id=57750
The 57546 bug looks very similar to what we are experiencing. We tested on
Linux and so far we do not see the same behavior. Also this is our web socket
code:
package ad.ecs.async.websocket;
import java.io.IOException;
import java.util.Arrays;
import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
import ad.common.Global;
import ad.ecs.async.AsyncEngine;
import ad.ecs.async.AsyncResponse;
import ad.ecs.async.AsyncType;
import ad.ecs.db.DatabaseEngine;
import ad.ecs.db.paradox.User;
import ad.ecs.security.engine.SecurityEngine;
/**
* @author Serge Perepel
* @since Aug 23, 2017 12:23:32 PM
*/
@ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)
public class ECSAsync {
@OnOpen
public void open(Session session) throws IOException{
session.getBasicRemote().sendText("Connection Established");
}
@OnMessage
public String login(String sessionID, Session session) {
AsyncEngine.INSTANCE.wsConnect(session, sessionID);
org.hibernate.Session dbSession =
DatabaseEngine.getSessionFactory().openSession();
try {
int userID =
SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);
User user = (User) dbSession.get(User.class, userID);
if (user != null) {
if (user.getNextLogin() == 1) {
AsyncResponse response = new AsyncResponse();
response.setType(AsyncType.None);
response.setData(Arrays.asList("PASSWORD"));
response.setObjData("PASSWORD");
AsyncEngine.INSTANCE.addTransientResult(sessionID, response);
}
}
} finally {
dbSession.close();
}
return "ok";
}
@OnClose
public void close(Session session, CloseReason reason) {
AsyncEngine.INSTANCE.wsDisconnect(session);
}
@OnError
public void error(Session session, Throwable error) {
Global.INSTANCE.getLogHelper().exception(error);
//session.close(new CloseReason(closeCode, reasonPhrase));
}
}
Front end opens connection and sends invalid sessionID which causes
the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw
exception and after that disconnect happens on the front end. At this point
front end opens new connection and process goes into the loop. I'm assuming
that the connection handler suppose to get freed after a disconnect happen. But
it seems to accumulate. You can try this code and just replace code in the
login method to always throw the exception. On the front end onDisconnect try
to open new connection and send a random message to the web socket.
Thank you
Serge
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]
Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
se...@american-data.com<mailto:se...@american-data.com> | www.american-data.com
[Follow us on
Facebook]<https://www.facebook.com/pages/American-Data/10044518233>
[Follow us on LinkedIn] <https://www.linkedin.com/company/american-data>
[Follow us on Twitter] <https://twitter.com/AmericanDataECS> [Follow us on
Twitter] <https://www.youtube.com/channel/UCiKfcsunZXWIYHf_2oE708A>
Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are
Suspected memory leak of org.apache.coyote.AbstractProtocol$ConnectionHandler object while using Websocket
When we run our app for 2-3 hours we experience this leak related to our web
sockets connections, here is the Memory Analyzer Suspect:
One instance of "org.apache.coyote.AbstractProtocol$ConnectionHandler" loaded
by "java.net.URLClassLoader @ 0x720029098" occupies 2,153,196,128 (88.10%)
bytes. The memory is accumulated in one instance of
"java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "".
Keywords
org.apache.coyote.AbstractProtocol$ConnectionHandler
java.util.concurrent.ConcurrentHashMap$Node[]
java.net.URLClassLoader @ 0x720029098
Also these bugs might be related to this one:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57546
https://bz.apache.org/bugzilla/show_bug.cgi?id=57750
The 57546 bug looks very similar to what we are experiencing. We tested on
Linux and so far we do not see the same behavior. Also this is our web socket
code:
package ad.ecs.async.websocket;
import java.io.IOException;
import java.util.Arrays;
import javax.websocket.CloseReason;
import javax.websocket.OnClose;
import javax.websocket.OnError;
import javax.websocket.OnMessage;
import javax.websocket.OnOpen;
import javax.websocket.Session;
import javax.websocket.server.ServerEndpoint;
import ad.common.Global;
import ad.ecs.async.AsyncEngine;
import ad.ecs.async.AsyncResponse;
import ad.ecs.async.AsyncType;
import ad.ecs.db.DatabaseEngine;
import ad.ecs.db.paradox.User;
import ad.ecs.security.engine.SecurityEngine;
/**
* @author Serge Perepel
* @since Aug 23, 2017 12:23:32 PM
*/
@ServerEndpoint(value = "/asyncMsg", encoders = AsyncResponseEncoder.class)
public class ECSAsync {
@OnOpen
public void open(Session session) throws IOException{
session.getBasicRemote().sendText("Connection Established");
}
@OnMessage
public String login(String sessionID, Session session) {
AsyncEngine.INSTANCE.wsConnect(session, sessionID);
org.hibernate.Session dbSession =
DatabaseEngine.getSessionFactory().openSession();
try {
int userID =
SecurityEngine.INSTANCE.getUserIDBasedOnSessionID(sessionID);
User user = (User) dbSession.get(User.class, userID);
if (user != null) {
if (user.getNextLogin() == 1) {
AsyncResponse response = new AsyncResponse();
response.setType(AsyncType.None);
response.setData(Arrays.asList("PASSWORD"));
response.setObjData("PASSWORD");
AsyncEngine.INSTANCE.addTransientResult(sessionID, response);
}
}
} finally {
dbSession.close();
}
return "ok";
}
@OnClose
public void close(Session session, CloseReason reason) {
AsyncEngine.INSTANCE.wsDisconnect(session);
}
@OnError
public void error(Session session, Throwable error) {
Global.INSTANCE.getLogHelper().exception(error);
//session.close(new CloseReason(closeCode, reasonPhrase));
}
}
Front end opens connection and sends invalid sessionID which causes
the line `AsyncEngine.INSTANCE.wsConnect(session, sessionID);` to throw
exception and after that disconnect happens on the front end. At this point
front end opens new connection and process goes into the loop. I'm assuming
that the connection handler suppose to get freed after a disconnect happen. But
it seems to accumulate. You can try this code and just replace code in the
login method to always throw the exception. On the front end onDisconnect try
to open new connection and send a random message to the web socket.
Thank you
Serge
[http://www.american-data.com/images/AmericanDataLogoRoundedEmailSignature.png]
Serge Perepel
Software Developer | American Data
p. 608.643.8022 | tf. 800.464.9942 | f. 608.643.2314
se...@american-data.com | www.american-data.com
[Follow us on
Facebook]
[Follow us on LinkedIn]
[Follow us on Twitter] [Follow us on
Twitter]
Notice: This message is the property of 1984 Systems, Inc. (DBA) American Data
and contains information that may be confidential and/or privileged. If you are
not
the intended recipient, you should not use, disclose or take any action based on
the message. If you have received this transmission in error, please immediately
contact the sender by return e-mail and delete this e-mail,
and any attachments, from any computer.
[ECS 10 Conversion]
