Hi Debraj, I think what Joakim told you is exactly what you need to do: start by removing all dependencies to commons-fileupload and work your way from there.
If you don`t, you'll end up wasting a lot of time only to create patchwork. Regards, *Nicolas Therrien* Senior Software Developer *[image: https://www.motorolasolutions.com/content/dam/msi/images/logos/corporate/msiemailsignature.png]* *o*: +1.819.931.2053 On Sat, Oct 13, 2018 at 12:19 PM Debraj Manna <subharaj.ma...@gmail.com> wrote: > Thanks Joakim for a detailed response. > > I don't think there is a problem in http-client as I can see that about > 3.5-3.8 GB out 5 GB is written to disk and then it gets stuck. I will > verify your other suggestions. But the problem is the issue does not come > always. It is intermittent. > > On Sat, Oct 13, 2018 at 5:49 PM Joakim Erdfelt <joa...@webtide.com> wrote: > >> The relevant stacktrace ... >> >> "dw-150 - POST /management/upgrade/uploadbundle" #150 prio=5 os_prio=0 >> tid=0x00007ffff0f7e800 nid=0x3a46 in Object.wait() [0x00007ffee0728000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> at java.lang.Object.wait(Object.java:502) >> at org.eclipse.jetty.server.HttpInput.blockForContent(HttpInput.java:565) >> at >> org.eclipse.jetty.server.HttpInput$1.blockForContent(HttpInput.java:1084) >> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:306) >> - locked <0x0000000677a5d220> (a java.util.ArrayDeque) >> at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999) >> at >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903) >> at java.io.InputStream.read(InputStream.java:101) >> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1488) >> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1465) >> at >> com.vnera.restapilayer.ManagementResource.flushFileToDisk(ManagementResource.java:2515) >> at >> com.vnera.restapilayer.ManagementResource.storeFileToDisk(ManagementResource.java:2441) >> at >> com.vnera.restapilayer.ManagementResource.uploadBundle(ManagementResource.java:2366) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at >> org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) >> at >> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) >> at >> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) >> at >> org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) >> at >> org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) >> at >> org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) >> at >> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) >> at >> org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) >> at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:315) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:297) >> at org.glassfish.jersey.internal.Errors.process(Errors.java:267) >> at >> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) >> at >> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) >> at >> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) >> at >> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) >> at >> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) >> at >> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) >> at >> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) >> at >> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) >> at >> io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:49) >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) >> at >> io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:34) >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) >> at >> io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45) >> at >> io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) >> at >> org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:308) >> at >> org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:262) >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) >> at >> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) >> at >> org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) >> at >> org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) >> at >> org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) >> at >> org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) >> at >> org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) >> at >> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) >> at >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >> at >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) >> at >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) >> at >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) >> at >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >> at >> com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:241) >> at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) >> at >> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:455) >> at io.dropwizard.jetty.BiDiGzipHandler.handle(BiDiGzipHandler.java:69) >> at >> org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) >> at >> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) >> at org.eclipse.jetty.server.Server.handle(Server.java:530) >> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) >> at >> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) >> at >> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) >> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) >> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) >> at >> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) >> at >> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) >> at >> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) >> at java.lang.Thread.run(Thread.java:748) >> >> Looks like you are using dropwizard + jersey + com.vnera.restapilayer + >> commons-fileupload to handle your file upload. >> >> Your setup isn't using Jetty for multipart/form-data upload in the way >> you think. >> >> First, don't use commons-fileupload, that's for Servlet 2.4 (and older) >> environments. >> Support for multipart/form-data parsing is built into the Servlet spec as >> HttpServletRequest.getParts() and getPart(name) since Servlet 3.0 >> See: >> https://docs.oracle.com/javaee/7/api/javax/servlet/http/HttpServletRequest.html#getPart-java.lang.String- >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.oracle.com_javaee_7_api_javax_servlet_http_HttpServletRequest.html-23getPart-2Djava.lang.String-2D&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=4cpaDgQbjItJZ6eUBOhoEvyZ5xcl-UEJyicVNSwKaQs&e=> >> >> If your code, or ANY of its Filters or 3rd party libraries uses any one >> of the HttpServletRequest.getParameter*() methods before your attempt at >> using commons-fileupload then your code will not work and fail. >> Why? That's because the Servlet spec says we have to read the Request >> URI query and Request form data (Input Stream) to generate the the >> parameter list and Parts list (if request is a multipart/form) >> >> Right now, based on your stacktrace, Jetty is only involved in giving you >> a HttpServletRequest InputStream which is blocked waiting for more content >> from the client. >> This could be a bug in commons-fileupload (a common scenario) >> A bug in your use of the servlet spec (based on the number of layers you >> have going on, this is a high probability, try putting a breakpoint on >> Jetty's org.eclipse.jetty.server.Request.getParameters() method to see if >> the above example is going on) >> Or a bug in your http client (a common scenario where the client says >> `Content-Length: <big-number>` but never sends the complete <big-number> of >> bytes, hence the blocking for content) >> >> You can test Jetty's behavior in isolation easily enough. >> >> $ mkdir jetty-downloads >> $ cd jetty-downloads >> $ curl -O >> http://central.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.12.v20180830/jetty-distribution-9.4.12.v20180830.tar.gz >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__central.maven.org_maven2_org_eclipse_jetty_jetty-2Ddistribution_9.4.12.v20180830_jetty-2Ddistribution-2D9.4.12.v20180830.tar.gz&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=_xubfGKK_hd_3C7SVWVrgcXmXZXXNcRILUISoABgvng&e=> >> $ tar -zxf tar -zxvf jetty-distribution-9.4.12.v20180830.tar.gz >> $ cd jetty-distribution-9.4.12.v20180830/demo-base/ >> $ java -jar ../start.jar >> >> now open a web browser to http://localhost:8080/test/dump/info >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8080_test_dump_info&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=djwkat47OBQtJYqNahL2TBoMQ0DKwuigizKTa_M-ktI&e=> >> scroll down to "Form to generate UPLOAD content" >> use "File 1" or "File 2" on that form to select your 5gb data file. >> then "submit" >> >> On chrome you'll see an "Uploading" percentage at the bottom of the >> window. >> >> when it's done uploading, you'll get a "Dump Servlet" response. >> scroll down to "Parts:" (about 1/3rd down the page) >> you'll see the results of HttpSerletRequest.getParts() here. >> >> example (for mine): >> Parts: >> Action: Part{n=Action,fn=null,ct=null,s=6,tmp=true,file=null} >> file2: >> Part{n=file2,fn=,ct=application/octet-stream,s=0,tmp=true,file=null} >> TextField: Part{n=TextField,fn=null,ct=null,s=7,tmp=true,file=null} >> file1: Part{n=file1,fn=bigger.iso,ct=application/octet-stream,s= >> *6806166040* >> ,tmp=true,file=/private/var/folders/gm/676mvm6j6131h37d0msqk1k40000gn/T/jetty-0.0.0.0-8080-test.war-_test-any-432009796042963862.dir/upload/MultiPart7642561650112679943} >> >> That tells me that 6806166040 bytes were sent as part of "file1". >> >> If I look at the original (pre-uploaded) file, I have ... >> >> $ ls -la bigger.iso >> -rw-r--r-- 1 joakim staff 6806166040 Oct 13 07:09 bigger.iso >> >> I can see that the upload completed, the entire 6GB file was sent. >> >> Joakim Erdfelt / joa...@webtide.com >> >> >> On Sat, Oct 13, 2018 at 6:17 AM Debraj Manna <subharaj.ma...@gmail.com> >> wrote: >> >>> POST /management/upgrade/uploadbundle is the api in context >>> >>> On Sat, Oct 13, 2018 at 4:26 PM Debraj Manna <subharaj.ma...@gmail.com> >>> wrote: >>> >>>> I am using a jetty-server behind nginx reverse proxy. On trying to >>>> upload a big file about 5 GB via Multi-part I am seeing sometimes the >>>> upload is getting stuck. Please find attached the full thread dump when the >>>> jetty server was stuck. >>>> >>>> I have also placed the thread dump in the below gist >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_debraj-2Dmanna_20b62dd3cbc9e7badb05fdb01e513fcd&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=hv6TjDnnkkwY-eeWVqCmgwMyVrNGLjbU6jABfarQVp4&e=> >>>> >>>> https://gist.github.com/debraj-manna/20b62dd3cbc9e7badb05fdb01e513fcd >>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_debraj-2Dmanna_20b62dd3cbc9e7badb05fdb01e513fcd&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=hv6TjDnnkkwY-eeWVqCmgwMyVrNGLjbU6jABfarQVp4&e=> >>>> >>>> Can someone let me know if this is some known issue in Jetty? >>>> >>>> Environment >>>> Ubuntu 14.04 >>>> Java 8 >>>> Jetty Server - 9.4.10, 9.4.8.v20171121 >>>> >>> _______________________________________________ >>> jetty-users mailing list >>> jetty-users@eclipse.org >>> To change your delivery options, retrieve your password, or unsubscribe >>> from this list, visit >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=> >> >> _______________________________________________ >> jetty-users mailing list >> jetty-users@eclipse.org >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=> > > _______________________________________________ > jetty-users mailing list > jetty-users@eclipse.org > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > > https://urldefense.proofpoint.com/v2/url?u=https-3A__dev.eclipse.org_mailman_listinfo_jetty-2Dusers&d=DwICAg&c=q3cDpHe1hF8lXU5EFjNM_A&r=P3_1pTtMQK06fFymYIWbyyzVU6nc0CcwfuZhLhexammvaiCaU0ieHeI7BWvfbbjE&m=v4n059clZKEoxfRB9GdzFxy5fdVagerkzRukYEqXuGY&s=Hpubhy2jdJWf-a4vF37_8cj4qCx0buzb-3pjfuFprfQ&e=
_______________________________________________ jetty-users mailing list jetty-users@eclipse.org To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
![https://www.motorolasolutions.com/content/dam/msi/images/logos/corporate/msiemailsignature.png]*](https://www.motorolasolutions.com/content/dam/msi/images/logos/corporate/msiemailsignature.png]*){kind=link}