Not sure if this is the case, but the "Connection: close" header cannot be added by Jetty if the HttpServletResponse.isCommitted() == true before using something like sendError().
Joakim Erdfelt / joa...@webtide.com On Wed, Sep 26, 2018 at 6:29 AM Tommy Becker <twbec...@gmail.com> wrote: > We definitely do not see one. But I'm still a bit confused as to how Jetty > is determining that it wants to close the connection. Although our JAX-RS > resource throws an exception, that exception should be handled by the > JAX-RS runtime and not propagated to Jetty (We have ExceptionMappers > defined for everything). So our application is generating the response and > not Jetty itself. > > On Wed, Sep 26, 2018 at 12:49 AM Greg Wilkins <gr...@webtide.com> wrote: > >> >> Thomas, >> >> I see a connection:close header when the connection is closed after >> sending a 400 or 500. >> >> However, nothing is sent if the connection is closed due to an abort >> while giving up reading unconsumed content, which can happen >> before/during/after a response hence we keep that simple. >> >> So are you sure you are seeing a 400/500 response without >> connection:close ? >> >> >> >> >> On Wed, 26 Sep 2018 at 14:33, Greg Wilkins <gr...@webtide.com> wrote: >> >>> >>> It is more about how the response was generated and less about the >>> response code itself. >>> If the application throws and exception to Jetty during request >>> handling, we now always make the connection non persistent before trying to >>> send a response. If the request input is terminated early or is not fully >>> consumed and would block, then we also abort the connection. >>> >>> Interesting that you say we don't set the Connection: close header. >>> There is actually no requirement to do so as the server can close a >>> connection at any time, but I thought we would do so as a courtesy.... >>> checking.... >>> >>> cheers >>> >>> >>> >>> On Wed, 26 Sep 2018 at 10:25, Tommy Becker <twbec...@gmail.com> wrote: >>> >>>> Thanks Greg. Just so I’m clear, what does Jetty key on to know whether >>>> to close the connection? Just the 4xx/5xx response code? I’m trying to >>>> understand the difference between this case and the “normal unconsumed >>>> input” case you describe. Also, I did notice that Jetty does not set the >>>> Connection: close header when it does this, is that intentional? >>>> >>>> >>>> On Sep 25, 2018, at 6:37 PM, Greg Wilkins <gr...@webtide.com> wrote: >>>> >>>> >>>> Thomas, >>>> >>>> There is no configuration to avoid this behaviour. If jetty sees and >>>> exception in the application it will send the 400 and close the connection. >>>> >>>> However, as Simone says, your application can be setup to avoid this >>>> situation by catching the exception and consuming any input. You can do >>>> this in a filter that catches Throwable, it can then check the request >>>> input stream (and/or reader) for unconsumed input and read & discard to end >>>> of file. If the response is not committed, it can then send a 400 or any >>>> other response that you like. >>>> >>>> Just remember that this may make your application somewhat vulnerable >>>> to DOS attacks as it will be easy to hold a thread in that filter slowly >>>> consuming data. I would suggest imposing a total time and total data limit >>>> on the input consumption. >>>> >>>> Note that for normal unconsumed input, jetty 9.4 does make some attempt >>>> to consume it... but if the reading of that data would block, it gives up >>>> and closes the connection, as there is no point blocking for data that will >>>> be discarded. >>>> >>>> regards >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, 26 Sep 2018 at 07:35, Thomas Becker <twbec...@gmail.com> wrote: >>>> >>>>> Thanks so much again for your response, this is great information. >>>>> What you say makes sense, but I now see I failed to mention the most >>>>> critical part of this problem. Which is that the client never actually >>>>> sees >>>>> the 400 response we are sending from Jetty. When varnish sees the RST, it >>>>> considers the backend request failed and returns 503 Service Unavailable >>>>> to >>>>> the client, effectively swallowing our application’s response. We can >>>>> pursue a solution to this on the Varnish side, but in the interim I’m >>>>> guessing there is no way to configure this behavior in Jetty? >>>>> >>>>> >>>>> >>>>> On Sep 25, 2018, at 4:28 PM, Simone Bordet <sbor...@webtide.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <twbec...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> Update: we setup an environment with the old Jetty 9.2 code and this >>>>> does not occur. 9.2 does not send the FIN in #5 above, and seems happy to >>>>> receive the rest of the content, despite having sent a response already. >>>>> >>>>> On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <twbec...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> Thanks for your response. I managed to snag a tcp dump of what's going >>>>> on in this scenario. From what I can see the sequence of events is the >>>>> following. Recall that our Jetty server is fronted by a Varnish cache. >>>>> >>>>> 1) Varnish sends the headers and initial part of the content for a >>>>> large POST. >>>>> 2) On the Jetty server, we use a streaming parser and begin validating >>>>> the content. >>>>> 3) We detect a problem with the content and throw an exception that >>>>> results in a 400 Bad Request to the client (via JAX-RS exception mapper) >>>>> 4) An ACK is sent for the segment containing the 400 error. >>>>> 5) The Jetty server sends a FIN. >>>>> 6) An ACK is sent for the FIN >>>>> 7) Varnish sends another segment that continues the content from #1. >>>>> 8) The Jetty server sends a RST. >>>>> >>>>> In the server logs, we see an Early EOF from our JAX-RS resource that >>>>> is parsing the content. This all seems pretty ok from the Jetty side, and >>>>> it certainly seems like Varnish is misbehaving here (I'm thinking it may >>>>> be >>>>> this bug https://github.com/varnishcache/varnish-cache/issues/2332). >>>>> But I'm still unclear as to why this started after our upgrade from Jetty >>>>> 9.2 -> 9.4. Any thoughts? >>>>> >>>>> >>>>> This is normal. >>>>> In Jetty 9.4 we are more aggressive in closing the connection because >>>>> we don't want to be at the mercy of a possible nasty client sending us >>>>> GiB of data when we know the application does not want to handle them. >>>>> Varnish behavior is correct too: it sees the FIN from Jetty but does >>>>> not know that Jetty does not want to read until it tries to send more >>>>> content and gets a RST. >>>>> At that point, it should relay the RST (or FIN) back to the client. >>>>> >>>>> So you have 2 choices: you catch the exception during your validation, >>>>> and finish to read (and discard) the content in the application; or >>>>> you ignore the early EOFs in the logs. >>>>> I don't think that those early EOFs are logged above DEBUG level, is >>>>> that correct? >>>>> >>>>> -- >>>>> Simone Bordet >>>>> ---- >>>>> http://cometd.org >>>>> http://webtide.com >>>>> Developer advice, training, services and support >>>>> from the Jetty & CometD experts. >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> jetty-users@eclipse.org >>>>> To change your delivery options, retrieve your password, or >>>>> unsubscribe from this list, visit >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>>> >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> jetty-users@eclipse.org >>>>> To change your delivery options, retrieve your password, or >>>>> unsubscribe from this list, visit >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>>> >>>> -- >>>> Greg Wilkins <gr...@webtide.com> CTO http://webtide.com >>>> _______________________________________________ >>>> jetty-users mailing list >>>> jetty-users@eclipse.org >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visit >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>>> _______________________________________________ >>>> jetty-users mailing list >>>> jetty-users@eclipse.org >>>> To change your delivery options, retrieve your password, or unsubscribe >>>> from this list, visit >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >>> >>> -- >>> Greg Wilkins <gr...@webtide.com> CTO http://webtide.com >>> >> >> >> -- >> Greg Wilkins <gr...@webtide.com> CTO http://webtide.com >> _______________________________________________ >> jetty-users mailing list >> jetty-users@eclipse.org >> To change your delivery options, retrieve your password, or unsubscribe >> from this list, visit >> https://dev.eclipse.org/mailman/listinfo/jetty-users > > _______________________________________________ > jetty-users mailing list > jetty-users@eclipse.org > To change your delivery options, retrieve your password, or unsubscribe > from this list, visit > https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________ jetty-users mailing list jetty-users@eclipse.org To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
