[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150629#comment-15150629 ] Mark Miller commented on SOLR-8453: --- Yeah, we are doing it in a filter, but we were just doing it on failures. I suspect some other code changes have exposed that we really should be doing it every request, but I've made that change. Currently, we don't really want the client or server to give up early in any case if we can help it. Since this issue is released, I filed a new JIRA for it, SOLR-8683 Always consume the full request on the server, not just in the case of an error. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, master > > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150173#comment-15150173 ] Greg Wilkins commented on SOLR-8453: If solr is in the habit of not consuming the content AND is particular about how such connections are closed/reused, then I believe a filter should be applied to all requests. I will have a look again to see if jetty can provide consumeAll behaviour as an option, but I really think that is just making solr dependent on a container features... as it was when it assumed the container would consume all of the input on it's behalf - which is behaviour not covered by any spec I know of. Other than that, you could make a feature request on the client to be able to handle request sending errors separately to response processing. Ie in this case may receive a valid response and the request body send still failed. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, master > > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149094#comment-15149094 ] Mark Miller commented on SOLR-8453: --- Seems only doing this on errors is not enough. I'm looking at upgrading Jetty again and still see a bunch of connection resets now unless I make sure to always fully consume the stream. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, master > > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097318#comment-15097318 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1724530 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1724530 ] SOLR-8453: Only consume input here if exp != null, otherwise it is done in writeResponse. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, Trunk > > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097316#comment-15097316 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1724529 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1724529 ] SOLR-8453: Only consume input here if exp != null, otherwise it is done in writeResponse. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 5.5, Trunk > > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096486#comment-15096486 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1724457 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1724457 ] SOLR-8453: Solr should attempt to consume the request inputstream on errors as we cannot count on the container to do it. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096467#comment-15096467 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1724450 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1724450 ] SOLR-8453: Solr should attempt to consume the request inputstream on errors as we cannot count on the container to do it. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095619#comment-15095619 ] Greg Wilkins commented on SOLR-8453: It depends on how the socket is closed. If the socket is RST rather than FIN'd then it is possible for the RST to overtake sent data - ie to cause some kind of exception with unconsumed data in a buffer. I have no idea if that is really the case or not. Jetty will to a proper close only if it has completed a HTTP message. Otherwise the close may be more brutal and may result in a RST (not directly under the control of java). > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094203#comment-15094203 ] Yonik Seeley commented on SOLR-8453: +1, looks fine. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, > jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089498#comment-15089498 ] Yonik Seeley commented on SOLR-8453: bq. I guess the HttpClient could potentially do better by making the status received in the response available, but then it is in a race because the close may occur prior to the response being read/parsed/processed. Not sure I understand this part. At the OS/socket level, the server can send the response and immediately close the socket, and the client (if written properly) can always read the response. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, > jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088866#comment-15088866 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1723660 from [~thetaphi] in branch 'dev/trunk' [ https://svn.apache.org/r1723660 ] SOLR-8453: Fix raw socket test to correctly use HTTP spec: headers in HTTP are US-ASCII, body's Content-Length is number of bytes (not chars!), PrintWriter swallows Exceptions and was not needed (Forbidden apis found the bug, but fix was incorrect) > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, > jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088691#comment-15088691 ] Mark Miller commented on SOLR-8453: --- bq. So if the client applications cannot be made more robust - ie it can't reasonably handle such connection failures without knowing the status code I don't think we can. As a user of HttpClient, we don't care about the connection reset. We do care about the error on the server though. We would want to be able to get the error, and ignore that we won't be reusing that connection. Solr wants connection reuse, but we don't want to be interrupted when it doesn't happen (especially if that also means we don't get the server error). I don't think we can get that option in the short term though, so best bet seems to be as you suggest in a filter. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, > jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088651#comment-15088651 ] Greg Wilkins commented on SOLR-8453: Ah - well I've evolved my view a bit, although the client (as in the part that uses the HttpClient) may need to be fixed. To repeat what I stated on the jetty-user list, I think both client and server are acting reasonably. Jetty sees an application return without having consumed all of a request - it assumes this is an error condition, makes a best effort attempt to consume, but when that requires blocking decides to close the connection before all the content is sent. The HttpClient is sending a chunked body and the connection is closed before it sends the terminal chunk, so apache is correct to return an exceptional condition to the client application, as the request did not end correctly. I guess the HttpClient could potentially do better by making the status received in the response available, but then it is in a race because the close may occur prior to the response being read/parsed/processed. So if the client applications cannot be made more robust - ie it can't reasonably handle such connection failures without knowing the status code, then the best solution is to avoid the server creating the ambiguity in the first place. If it is important for the client that all the content is consumed, then the server application should consume all the content, even if there is an error.A filter can achieve that or perhaps a sendError wrapper. We really don't want Jetty to have to do this consumeAll, because we then cannot protect asynchronous apps with small thread pools from DOS attacks. cheers > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088506#comment-15088506 ] Mark Miller commented on SOLR-8453: --- To expand: If HttpClient is behaving reasonably by throwing an exception about connection reset, Solr cannot get the actual server exception, which is essential to our proper operation. I don't see what we can do as a client without changes to HttpClient. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088501#comment-15088501 ] Mark Miller commented on SOLR-8453: --- bq. But the client should still be fixed IMHO. On the Jetty mailing list you mentioned that Apache HttpClient seemed to be acting reasonably. I'm not sure how that relates to this comment about the client being fixed? > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088495#comment-15088495 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1723646 from [~tomasflobbe] in branch 'dev/trunk' [ https://svn.apache.org/r1723646 ] SOLR-8453: Fix precommit > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088394#comment-15088394 ] Greg Wilkins commented on SOLR-8453: Note I just added a comment on 7339 - that this can be worked around by adding a filter that consumes all the content. But the client should still be fixed IMHO. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088327#comment-15088327 ] Greg Wilkins commented on SOLR-8453: Copying this comment over from the jetty-users discussion (and expanding somewhat) This is indeed a change in behaviour in jetty 9.3, but not one that should affect any application. The problem with consumeAll previously is that it was an unlimited commitment - a client could send chunks forever and consume a thread forever. The only reason we need to consume all is to maintain a persistent connection. But a server is under no obligation to maintain a persistent connection (even if the application consumes all the request data). So if jetty doesn't consumeAll, we can close that connection and the client needs to start a new one. So the approach that 9.3 takes is that if the application has not read all it's content (which is normally an error condition of the application), then jetty will make a best effort attempt to read the content, but only if it can do so without blocking. If it has to block - it gives up and the connection is closed. The decision is that it is better to throw away the connection rather than commit to blocking for an unknown period of time. This is more important now as apps are using the async APIs and are configured with minimal threadpools - thus we need to avoid places where the container can block and consume the thread pool (and thus be vulnerable to DOS attacks). So your client needs to be more robust in this circumstance and/or your application should make a better attempt to consume all the data. Even if we reinstated the behaviour in jetty - it would really just be happenstance that your client works and intermediaries/proxies could change the behaviour. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087912#comment-15087912 ] ASF subversion and git services commented on SOLR-8453: --- Commit 1723613 from [~yo...@apache.org] in branch 'dev/trunk' [ https://svn.apache.org/r1723613 ] SOLR-8453: test HTTP client error responses > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch, jetty9.2.pcapng, jetty9.3.pcapng > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[ https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087662#comment-15087662 ] Mark Miller commented on SOLR-8453: --- I think someone reached out to the Jetty project in SOLR-7339. I'll attach a wireshark log I have. Wireshark and debug logs all confirm this change in behavior. > Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate > client connections. > --- > > Key: SOLR-8453 > URL: https://issues.apache.org/jira/browse/SOLR-8453 > Project: Solr > Issue Type: Bug >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, > SOLR-8453_test.patch, SOLR-8453_test.patch > > > The basic problem is that when we are streaming in updates via a client, an > update can fail in a way that further updates in the request will not be > processed, but not in a way that causes the client to stop and finish up the > request before the server does something else with that connection. > This seems to mean that even after the server stops processing the request, > the concurrent update client is still in the process of sending the request. > It seems previously, Jetty would not go after the connection very quickly > after the server processing thread was stopped via exception, and the client > (usually?) had time to clean up properly. But after the Jetty upgrade from > 9.2 to 9.3, Jetty closes the connection on the server sooner than previous > versions (?), and the client does not end up getting notified of the original > exception at all and instead hits a connection reset exception. The result > was random fails due to connection reset throughout our tests and one > particular test failing consistently. Even before this update, it does not > seem like we are acting in a safe or 'behaved' manner, but our version of > Jetty was relaxed enough (or a bug was fixed?) for our tests to work out. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[
https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087580#comment-15087580
]
Yonik Seeley commented on SOLR-8453:
Nice find!
I've been messing around on the client side with HttpURLConnection and raw
sockets.
Even with HttpURLConnection (the stuff built into the JVM) I can't even get the
status code...
The server has probably written the error response, but the client side wants
to push the complete request before looking at it I guess:
{code}
Caused by: java.io.IOException: Error writing to server
at
sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:665)
at
sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:677)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1533)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at
org.apache.solr.client.solrj.TestSolrJErrorHandling.testHttpURLConnection(TestSolrJErrorHandling.java:242)
{code}
Catching the exception and trying to get the errorStream also doesn't work.
Anyway, I had a thought that even if we could fix this in the SolrJ clients,
there are a lot of other clients out there that will get bitten by this.
Regardless of what improvements we make to SolrJ, it seems like we should roll
back the Jetty upgrade.
> Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate
> client connections.
> ---
>
> Key: SOLR-8453
> URL: https://issues.apache.org/jira/browse/SOLR-8453
> Project: Solr
> Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch,
> SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch,
> SOLR-8453_test.patch, SOLR-8453_test.patch
>
>
> The basic problem is that when we are streaming in updates via a client, an
> update can fail in a way that further updates in the request will not be
> processed, but not in a way that causes the client to stop and finish up the
> request before the server does something else with that connection.
> This seems to mean that even after the server stops processing the request,
> the concurrent update client is still in the process of sending the request.
> It seems previously, Jetty would not go after the connection very quickly
> after the server processing thread was stopped via exception, and the client
> (usually?) had time to clean up properly. But after the Jetty upgrade from
> 9.2 to 9.3, Jetty closes the connection on the server sooner than previous
> versions (?), and the client does not end up getting notified of the original
> exception at all and instead hits a connection reset exception. The result
> was random fails due to connection reset throughout our tests and one
> particular test failing consistently. Even before this update, it does not
> seem like we are acting in a safe or 'behaved' manner, but our version of
> Jetty was relaxed enough (or a bug was fixed?) for our tests to work out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8453) Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate client connections.
[
https://issues.apache.org/jira/browse/SOLR-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086065#comment-15086065
]
Mark Miller commented on SOLR-8453:
---
Yup, seems to go away with this hack:
{code}
public boolean consumeAll()
{
synchronized (_inputQ)
{
try
{
while (!isFinished())
{
Content item = nextContent();
if (item == null) {
blockForContent();
item = nextContent();
}
if (item != null) skip(item, remaining(item));
}
return true;
}
catch (IOException e)
{
LOG.debug(e);
return false;
}
}
}
{code}
> Jetty update from 9.2 to 9.3 causes the server to reset formerly legitimate
> client connections.
> ---
>
> Key: SOLR-8453
> URL: https://issues.apache.org/jira/browse/SOLR-8453
> Project: Solr
> Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch,
> SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch, SOLR-8453.patch,
> SOLR-8453_test.patch, SOLR-8453_test.patch
>
>
> The basic problem is that when we are streaming in updates via a client, an
> update can fail in a way that further updates in the request will not be
> processed, but not in a way that causes the client to stop and finish up the
> request before the server does something else with that connection.
> This seems to mean that even after the server stops processing the request,
> the concurrent update client is still in the process of sending the request.
> It seems previously, Jetty would not go after the connection very quickly
> after the server processing thread was stopped via exception, and the client
> (usually?) had time to clean up properly. But after the Jetty upgrade from
> 9.2 to 9.3, Jetty closes the connection on the server sooner than previous
> versions (?), and the client does not end up getting notified of the original
> exception at all and instead hits a connection reset exception. The result
> was random fails due to connection reset throughout our tests and one
> particular test failing consistently. Even before this update, it does not
> seem like we are acting in a safe or 'behaved' manner, but our version of
> Jetty was relaxed enough (or a bug was fixed?) for our tests to work out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
