[ https://issues.apache.org/jira/browse/TS-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Jackson resolved TS-3959. -------------------------------- Resolution: Fixed > Dropped keep-alive connections not being re-established > ------------------------------------------------------- > > Key: TS-3959 > URL: https://issues.apache.org/jira/browse/TS-3959 > Project: Traffic Server > Issue Type: Bug > Components: Core, Network > Affects Versions: 6.0.0 > Reporter: Nick Muerdter > Assignee: Thomas Jackson > Priority: Blocker > Labels: regression > Fix For: 7.0.0 > > Attachments: trafficserver-closed.png, trafficserver-reset.png > > > I've observed some differences in how TrafficServer 6.0.0 behaves with > connection retrying and outgoing keep-alive connections. I believe the > changes in behavior might be related to this issue: > https://issues.apache.org/jira/browse/TS-3440 > I originally wasn't sure if this was a bug, but James Peach indicated it > sounded more like a regression on the mailing list > (http://mail-archives.apache.org/mod_mbox/trafficserver-users/201510.mbox/%3cba85d5a2-8b29-44a9-acdc-e7fa8d21f...@apache.org%3e). > What I'm seeing in 6.0.0 is that if TrafficServer has some backend keep-alive > connections already opened, but then one of the keep-alive connections is > closed, the next request to TrafficServer may generate a 502 Server Hangup > response when attempting to reuse that connection. Previously, I think > TrafficServer was retrying when it encountered a closed keep-alive > connection, but that is no longer the case. So if you have a backend that > might unexpectedly close its open keep-alive connections, the only way I've > found to completely prevent these 502 errors in 6.0.0 is to disable outgoing > keepalive (proxy.config.http.keep_alive_enabled_out and > proxy.config.http.keep_alive_post_out settings). > For a slightly more concrete example of what can trigger this, this is fairly > easy to reproduce with the following setup: > - TrafficServer is proxying to nginx with outgoing keep-alive connections > enabled (the default). > - Throw a constant stream of requests at TrafficServer. > - While that constant stream of requests is happening, also send a regular > stream of SIGHUP commands to nginx to reload nginx. > - Eventually you'll get some 502 Server Hangup responses from TrafficServer > among your stream of requests. > SIGHUPs in nginx should result in zero downtime for new requests, but I think > what's happening is that TrafficServer may fail when an old keep-alived > connection is reused (it's not common, so it depends on the timing of things > and if the connection is from an old nginx worker that has since been shut > down). In TrafficServer 5.3.1 these connection failures were retried, but in > 6.0.0, no retries occur in this case. > Here's some debug logs that show the difference in behavior between 6.0.0 and > 5.3.1. Note that differences seem to stem from how each version eventually > handles the "VC_EVENT_EOS" event following > "&HttpSM::state_send_server_request_header, VC_EVENT_WRITE_COMPLETE". > 5.3.1: > https://gist.github.com/GUI/0c53a6c4fdc2782b14aa#file-trafficserver_5-3-1-log-L316 > 6.0.0: > https://gist.github.com/GUI/0c53a6c4fdc2782b14aa#file-trafficserver_6-0-0-log-L314 > Interestingly, if I'm understand the log files correctly, it looks like > TraffficServer is reporting an odd empty response from these connections > ("HTTP/0.9 0" in 5.3.1 and "HTTP/1.0 0" in 6.0.0). However, as far as I can > tell from TCP dumps on the system, nginx is not actually sending any form of > response. > In these example cases the backend server isn't sending back any data (at > least as far as I can tell), so from what I understand (and the logic > outlined in https://issues.apache.org/jira/browse/TS-3440), it should be safe > to retry. > Let me know if I can provide any other details. Or if exact scripts to > reproduce the issues against the example nginx backend I described above > would be useful, I could get that together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)