Thanks Joakim for a detailed response. I don't think there is a problem in http-client as I can see that about 3.5-3.8 GB out 5 GB is written to disk and then it gets stuck. I will verify your other suggestions. But the problem is the issue does not come always. It is intermittent.
On Sat, Oct 13, 2018 at 5:49 PM Joakim Erdfelt <joa...@webtide.com> wrote: > The relevant stacktrace ... > > "dw-150 - POST /management/upgrade/uploadbundle" #150 prio=5 os_prio=0 > tid=0x00007ffff0f7e800 nid=0x3a46 in Object.wait() [0x00007ffee0728000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at org.eclipse.jetty.server.HttpInput.blockForContent(HttpInput.java:565) > at > org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput.java:1084) > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:306) > - locked <0x0000000677a5d220> (a java.util.ArrayDeque) > at > org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999) > at > org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903) > at java.io.InputStream.read(InputStream.java:101) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488) > at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1465) > at > com.vnera.restapilayer.ManagementResource.flushFileToDisk(ManagementResource.java:2515) > at > com.vnera.restapilayer.ManagementResource.storeFileToDisk(ManagementResource.java:2441) > at > com.vnera.restapilayer.ManagementResource.uploadBundle(ManagementResource.java:2366) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) > at > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) > at > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) > at > org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) > at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > at > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > at > org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) > at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) > at > io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) > at > io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) > at > io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45) > at > io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) > at > org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) > at > org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > at > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) > at > org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at > com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241) > at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) > at > org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455) > at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69) > at > org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) > at > org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) > at org.eclipse.jetty.server.Server.handle(Server.java:530) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) > at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) > at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) > at > org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) > at > org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) > at java.lang.Thread.run(Thread.java:748) > > Looks like you are using dropwizard + jersey + com.vnera.restapilayer + > commons-fileupload to handle your file upload. > > Your setup isn't using Jetty for multipart/form-data upload in the way you > think. > > First, don't use commons-fileupload, that's for Servlet 2.4 (and older) > environments. > Support for multipart/form-data parsing is built into the Servlet spec as > HttpServletRequest.getParts() and getPart(name) since Servlet 3.0 > See: > https://docs.oracle.com/javaee/7/api/javax/servlet/http/HttpServletRequest.html#getPart-java.lang.String- > > If your code, or ANY of its Filters or 3rd party libraries uses any one of > the HttpServletRequest.getParameter*() methods before your attempt at using > commons-fileupload then your code will not work and fail. > Why? That's because the Servlet spec says we have to read the Request URI > query and Request form data (Input Stream) to generate the the parameter > list and Parts list (if request is a multipart/form) > > Right now, based on your stacktrace, Jetty is only involved in giving you > a HttpServletRequest InputStream which is blocked waiting for more content > from the client. > This could be a bug in commons-fileupload (a common scenario) > A bug in your use of the servlet spec (based on the number of layers you > have going on, this is a high probability, try putting a breakpoint on > Jetty's org.eclipse.jetty.server.Request.getParameters() method to see if > the above example is going on) > Or a bug in your http client (a common scenario where the client says > `Content-Length: <big-number>` but never sends the complete <big-number> of > bytes, hence the blocking for content) > > You can test Jetty's behavior in isolation easily enough. > > $ mkdir jetty-downloads > $ cd jetty-downloads > $ curl -O > http://central.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.12.v20180830/jetty-distribution-9.4.12.v20180830.tar.gz > $ tar -zxf tar -zxvf jetty-distribution-9.4.12.v20180830.tar.gz > $ cd jetty-distribution-9.4.12.v20180830/demo-base/ > $ java -jar ../start.jar > > now open a web browser to http://localhost:8080/test/dump/info > scroll down to "Form to generate UPLOAD content" > use "File 1" or "File 2" on that form to select your 5gb data file. > then "submit" > > On chrome you'll see an "Uploading" percentage at the bottom of the window. > > when it's done uploading, you'll get a "Dump Servlet" response. > scroll down to "Parts:" (about 1/3rd down the page) > you'll see the results of HttpSerletRequest.getParts() here. > > example (for mine): > Parts: > Action: Part{n=Action,fn=null,ct=null,s=6,tmp=true,file=null} > file2: > Part{n=file2,fn=,ct=application/octet-stream,s=0,tmp=true,file=null} > TextField: Part{n=TextField,fn=null,ct=null,s=7,tmp=true,file=null} > file1: Part{n=file1,fn=bigger.iso,ct=application/octet-stream,s= > *6806166040* > ,tmp=true,file=/private/var/folders/gm/676mvm6j6131h37d0msqk1k40000gn/T/jetty-0.0.0.0-8080-test.war-_test-any-432009796042963862.dir/upload/MultiPart7642561650112679943} > > That tells me that 6806166040 bytes were sent as part of "file1". > > If I look at the original (pre-uploaded) file, I have ... > > $ ls -la bigger.iso > -rw-r--r-- 1 joakim staff 6806166040 Oct 13 07:09 bigger.iso > > I can see that the upload completed, the entire 6GB file was sent. > > Joakim Erdfelt / joa...@webtide.com > > > On Sat, Oct 13, 2018 at 6:17 AM Debraj Manna <subharaj.ma...@gmail.com> > wrote: > >> POST /management/upgrade/uploadbundle is the api in context >> >> On Sat, Oct 13, 2018 at 4:26 PM Debraj Manna <subharaj.ma...@gmail.com> >> wrote: >> >>> I am using a jetty-server behind nginx reverse proxy. On trying to >>> upload a big file about 5 GB via Multi-part I am seeing sometimes the >>> upload is getting stuck. Please find attached the full thread dump when the >>> jetty server was stuck. >>> >>> I have also placed the thread dump in the below gist >>> <https://gist.github.com/debraj-manna/20b62dd3cbc9e7badb05fdb01e513fcd> >>> >>> https://gist.github.com/debraj-manna/20b62dd3cbc9e7badb05fdb01e513fcd >>> >>> Can someone let me know if this is some known issue in Jetty? >>> >>> Environment >>> Ubuntu 14.04 >>> Java 8 >>> Jetty Server - 9.4.10, 9.4.8.v20171121 >>> >> _______________________________________________ >> jetty-users mailing list >> jetty-users@eclipse.org >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users > > _______________________________________________ > jetty-users mailing list > jetty-users@eclipse.org > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________ jetty-users mailing list jetty-users@eclipse.org To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
