Re: Backend polling not working
In message <4aded57c.3020...@upfrontsystems.co.za>, Izak Burger writes: >Hello again, > >I've discussed this issue on zope-dev. The agreement seems to be that >there is some ambiguity in the HTTP1.1 specification, but that the >general accepted interpretation is that if the other end closes the >connection you should close your end as well (it signals a timeout). > >There is also no browser implementation out there that does half-closing >(except where SSL is involved). None that I know of anyway. I wasn't quite sure which way to interpret the text originally so I did it the way that was most convenient for the code. I'll change it. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
Hello again, I've discussed this issue on zope-dev. The agreement seems to be that there is some ambiguity in the HTTP1.1 specification, but that the general accepted interpretation is that if the other end closes the connection you should close your end as well (it signals a timeout). There is also no browser implementation out there that does half-closing (except where SSL is involved). None that I know of anyway. https://mail.zope.org/pipermail/zope-dev/2009-October/038133.html This also affects nginx. I would therefore like to submit a wishlist item to make this configurable. I've already hacked the offending shutdown() call out of the code to test, and it works perfectly with zope once you do that. If there are no objections, can I go ahead and log this as a wish/feature request on the bugtracker? regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
Apologies for the (mostly) monologue this far :-) Izak Burger wrote: > Alright, I have to take that back. I changed the port so varnish talks > to apache instead and that works. That means zope somehow manages to > cause a POLLIN event on the file descriptor even though there is nothing > to read. Will have to do more research on this. It appears the fault is somewhere in zope. I ended up writing a test program to illustrate the problem. The reason why poll() returns and yet read() has nothing to read is that poll() always implicitly checks for the additional events POLLERR, POLLHUP and POLLLNVAL, so simply checking that poll() returns greater than zero is not a guarantee that there is data to be read. In my case, poll() returns immediately because POLLHUP has occured. It seems that zope reacts (incorrectly it would seem) on the shutdown() call and closes down the entire connection. I will have to take this up with zope-dev. regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
Izak Burger wrote: > I then added some extra log statements to the C code and confirmed that > the do/while loop that is supposed to read the response terminates on > the very first read() call with an EOF. > > In other words, there is definitely something wrong with varnish here. > > I suspect the problem might be with poll(), although I have yet to > confirm that. Specifically, it seems that the case where poll() returns > -1 is not handled properly. At the least I think it should handle EINTR? > I'll check that later and let you know. Alright, I have to take that back. I changed the port so varnish talks to apache instead and that works. That means zope somehow manages to cause a POLLIN event on the file descriptor even though there is nothing to read. Will have to do more research on this. From various sources found by google, it seems backend probing fails in a similar manner with nginx. regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
Izak Burger wrote: > I had the same thought, so I took the request as captured with the > packet grabber and tried it against zope using telnet, and it produces > the required result. That in itself is perhaps a hint, I think the > telnet client inserts the correct CRLF sequences which might not be > immediately apparent. > > Also, if read(2) returns zero it means EOF, so the problem must somehow > be on the backend. I'll reply again on this thread if I manage to figure > it out. When inspecting the network traffic I noticed that the client side (varnish) sends the first FIN packet, and not the server/backend (zope). This means that varnish closes the connection without reading the response. As far as I know, when you specify "Connection: close", the server will close the connection after the response rather than the other way round. I then added some extra log statements to the C code and confirmed that the do/while loop that is supposed to read the response terminates on the very first read() call with an EOF. In other words, there is definitely something wrong with varnish here. I suspect the problem might be with poll(), although I have yet to confirm that. Specifically, it seems that the case where poll() returns -1 is not handled properly. At the least I think it should handle EINTR? I'll check that later and let you know. regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
Poul-Henning Kamp wrote: > Yes, if we get no bytes, we do not accept it as a good probe. Yes, saw that in the svn logs, and I agree with it. > You may need to customize the HTTP request used for probing with > the ".request =" line in your backend probe definition, in order > to explain things to your backend. A Host: header maybe ? I had the same thought, so I took the request as captured with the packet grabber and tried it against zope using telnet, and it produces the required result. That in itself is perhaps a hint, I think the telnet client inserts the correct CRLF sequences which might not be immediately apparent. Also, if read(2) returns zero it means EOF, so the problem must somehow be on the backend. I'll reply again on this thread if I manage to figure it out. regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Re: Backend polling not working
In message <4ad60d6d.1070...@upfrontsystems.co.za>, Izak Burger writes: >Hi all, > >10.0.0.12 - Anonymous [14/Oct/2009:18:38:38 +0200] "GET /plonesite >HTTP/1.1" 200 0 "" > >Note the zero after the 200 status. I suspect this causes a problem on >line 220 of bin/varnishd/cache_backend_poll.c, where rlen is the length >of data read from zope: > > if (rlen == 0) > return; > >The result of this is a sick backend, eg: Yes, if we get no bytes, we do not accept it as a good probe. >Using wireshark to log at the network traffic confirms that the HTTP >request is sent, but no response, corresponding with the access log. That would sort of indicate that the problem is on the backend. You may need to customize the HTTP request used for probing with the ".request =" line in your backend probe definition, in order to explain things to your backend. A Host: header maybe ? Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
Backend polling not working
Hi all, I tried varnish today for the first time. This happened after we upgraded to a newer version of squid and it hung itself in an infinite loop. But that is a different story. We want to use varnish to front a number of zope backends, all of which speak to the same database backend (so they serve the same content). The problem with this zope application is that it takes a long time to reload. Those who know plone knows it is very heavy on the catalog, and in our case it means that after restarting a backend, it can take up to 15 minutes for the caches to warm up again. During this time, we would like the zope backends to be marked as sick and not be used. Unfortunately this does not seem to work in version 2.0.4 (which seems the most recent if I check your svn tags). The odd thing is this: it seems that the probe succeeds, but zope's access log shows that zero bytes is transferred, eg: 10.0.0.12 - Anonymous [14/Oct/2009:18:38:38 +0200] "GET /plonesite HTTP/1.1" 200 0 "" Note the zero after the 200 status. I suspect this causes a problem on line 220 of bin/varnishd/cache_backend_poll.c, where rlen is the length of data read from zope: if (rlen == 0) return; The result of this is a sick backend, eg: 0 Backend_health - live1 Still sick 4--X-S--- 0 2 3 0.00 0.00 Using wireshark to log at the network traffic confirms that the HTTP request is sent, but no response, corresponding with the access log. This seems very similar to this report: http://projects.linpro.no/pipermail/varnish-dev/2009-February/002086.html Has anyone seen this problem before? Perhaps there is a fix in svn already? I could not see anything that looked promising while looking through the subversion logs. regards, Izak ___ varnish-dev mailing list varnish-dev@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-dev
