Hello jetty-users,
My application is an image/video storage service, which stores/serves up to
100 mb files (traffic is low, under 5 QPS). The uploaded files are stored
on Google Cloud Storage. My server runs on jetty-9.2.13.v20150730.
Native Memory Leak:
I noticed a slow native memory leak in my application. It starts with 3G
memory, then gradually grows to 6G after 10 days, and the gradually grows
to 9G after another 10 days. I took a heap dump but the could not find
anything significant, heap size is around 1.5G.
Early EOF:
Besides the native memory leak, another noticeable problem is that I am
getting "Early EOF" exceptions from time to time (*full* stacktrace in the
end). I was monitoring the TCP traffic, and noticed that this exception is
thrown when the client sends Fin to server, after which server tries to
push more data to client and client sends back a Reset. (tcpdump snippet in
the end)
Question:
So, I am seeking guidance about:
1. Could the Reset package be the cause of the Early EOF? How should I
handle it properly?
2. Any suggestions on the direction of investigating the native memory leak?
3. Could these two issues be related to each other?
TCPDump snippet:
client > server: Flags [P.], ... <= client pushes some final data
server > client: Flags [.], ... <= server acks on the data
client > server: Flags [F.], ... <= client sends Fin Ack
server > client: Flags [.], ... <= server Ack the Fin
server > client: Flags [P.], ... <= server tries to push more data
client > server: Flags [R], ... <= client sends Reset
Stacktrace:
org.eclipse.jetty.io.EofException: Early EOF
at
org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:506)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:124)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:943)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:922)
at
com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:426)
at
com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:427)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at MyController.uploadStream(MyController.java:1111)
at MyController.handlePost(MyController.java:596)
at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
_______________________________________________
jetty-users mailing list
[email protected]
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
