Re: 7.3 + kqueue + apache/php + DNS lookup problem
On 1 October 2011 03:18, Doug Barton do...@freebsd.org wrote: It's a php module doing a lookup for the hostname of the back-end mysql server. Are the delays always 3 seconds? Pretty much. If so, that almost sounds like a timeout of some kind. That was my first thought, but the answer always comes eventually. To answer Chuck's questions, no threading is involved, and it's not apache doing the lookups. Doug Check your bind/unbound logs to ensure the queries are actually successful on their first try. Is your DNS using forwarders ? views ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
On Sun, Oct 02, 2011 at 10:18:30AM +0200, Damien Fleuriot wrote: On 1 October 2011 03:18, Doug Barton do...@freebsd.org wrote: It's a php module doing a lookup for the hostname of the back-end mysql server. Are the delays always 3 seconds? Pretty much. If so, that almost sounds like a timeout of some kind. That was my first thought, but the answer always comes eventually. To answer Chuck's questions, no threading is involved, and it's not apache doing the lookups. Doug Check your bind/unbound logs to ensure the queries are actually successful on their first try. Is your DNS using forwarders ? views ? How would this explain 100% quick/reliable lookups when done from tools like nslookup and host? Same box and same resolver (127.0.0.1:53), yet different behaviour (nslookup/host vs. PHP). -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
Something stupid and ridiculous, like the socket watermark points are set incorrectly? It'd also be helpful to see exactly what the knotes were. Adrian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
On 01/10/2011 02:18, Doug Barton wrote: Does this happen when httpd tries to do DNS resolution for, say, an incoming connection to the web server (e.g. trying to resolve the incoming IP address of the client to an FQDN), or is it happening within some PHP code (assuming PHP is installed/used as an Apache module) that's trying to do DNS resolution of some kind? It's a php module doing a lookup for the hostname of the back-end mysql server. Hmmm... Is this a function of DNS traffic being via UDP? Presumably you're not seeing the same sort of delays when eg. apache connects to mysql via TCP. Hard to think of another UDP protocol you could use to test -- SNMP perhaps? Or somehow forcing the DNS traffic to go via TCP? Tricky to make that happen when the resolver is on localhost. Of course, since DNS will only fall back to TCP after trying UDP, that's going to be even slower overall than your current situation, but the point here is to examine the truss output for timing details specifically around where the TCP query is issued. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matt...@infracaninophile.co.uk Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: 7.3 + kqueue + apache/php + DNS lookup problem
On 10/02/2011 12:10, Matthew Seaman wrote: On 01/10/2011 02:18, Doug Barton wrote: Does this happen when httpd tries to do DNS resolution for, say, an incoming connection to the web server (e.g. trying to resolve the incoming IP address of the client to an FQDN), or is it happening within some PHP code (assuming PHP is installed/used as an Apache module) that's trying to do DNS resolution of some kind? It's a php module doing a lookup for the hostname of the back-end mysql server. Hmmm... Is this a function of DNS traffic being via UDP? Presumably you're not seeing the same sort of delays when eg. apache connects to mysql via TCP. Hard to think of another UDP protocol you could use to test -- SNMP perhaps? Or somehow forcing the DNS traffic to go via TCP? Tricky to make that happen when the resolver is on localhost. Of course, since DNS will only fall back to TCP after trying UDP, that's going to be even slower overall than your current situation, but the point here is to examine the truss output for timing details specifically around where the TCP query is issued. Cheers, Matthew What is the exact query issued and what was the response? I see recvfrom returned 30 bytes in Doug's original mail which seems awfully short for a meaningful DNS response. Cheers Michiel ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
7.3 + kqueue + apache/php + DNS lookup problem
Howdy,
So, this is a bit of an odd one I've got a web server running
apache 2.2.17 and php 5.3.5. The host itself is running 7.3-RELEASE,
i386, and is not busy. I can do DNS queries on the command line all day
long and they are very snappy. Using nslookup, dig, whatever.
The weirdness comes in when the httpd process needs to do a DNS lookup.
With a local cache I'm getting the following:
97625 httpd0.031139 CALL connect(0x55,0x284fd590,0x10)
97625 httpd0.031142 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd0.031150 RET connect 0
97625 httpd0.031153 CALL sendto(0x55,0x2a7d1000,0x1e,0,0,0)
97625 httpd0.031169 GIO fd 85 wrote 30 bytes
97625 httpd0.031173 RET sendto 30/0x1e
97625 httpd0.031179 CALL clock_gettime(0,0xbfbfb58c)
97625 httpd0.031184 RET clock_gettime 0
97625 httpd0.031188 CALL
kevent(0x54,0xbfbfb678,0x1,0xbfbfb678,0x1,0xbfbfb68c)
97625 httpd3.064266 GIO fd 84 wrote 20 bytes
note the 3 second delay here.
97625 httpd3.064277 GIO fd 84 read 20 bytes
97625 httpd3.064281 RET kevent 1
97625 httpd3.064287 CALL
recvfrom(0x55,0x2a6c4000,0x1,0,0xbfbfb5f8,0xbfbfb694)
97625 httpd3.064293 GIO fd 85 read 30 bytes
97625 httpd3.064296 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd3.064299 RET recvfrom 30/0x1e
97625 httpd3.064308 CALL close(0x55)
I'm open to suggestions on where to look to improve this situation.
Thanks,
Doug
--
Nothin' ever doesn't change, but nothin' changes much.
-- OK Go
Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
On Fri, Sep 30, 2011 at 04:31:18PM -0700, Doug Barton wrote:
Howdy,
So, this is a bit of an odd one I've got a web server running
apache 2.2.17 and php 5.3.5. The host itself is running 7.3-RELEASE,
i386, and is not busy. I can do DNS queries on the command line all day
long and they are very snappy. Using nslookup, dig, whatever.
The weirdness comes in when the httpd process needs to do a DNS lookup.
With a local cache I'm getting the following:
97625 httpd0.031139 CALL connect(0x55,0x284fd590,0x10)
97625 httpd0.031142 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd0.031150 RET connect 0
97625 httpd0.031153 CALL sendto(0x55,0x2a7d1000,0x1e,0,0,0)
97625 httpd0.031169 GIO fd 85 wrote 30 bytes
97625 httpd0.031173 RET sendto 30/0x1e
97625 httpd0.031179 CALL clock_gettime(0,0xbfbfb58c)
97625 httpd0.031184 RET clock_gettime 0
97625 httpd0.031188 CALL
kevent(0x54,0xbfbfb678,0x1,0xbfbfb678,0x1,0xbfbfb68c)
97625 httpd3.064266 GIO fd 84 wrote 20 bytes
note the 3 second delay here.
97625 httpd3.064277 GIO fd 84 read 20 bytes
97625 httpd3.064281 RET kevent 1
97625 httpd3.064287 CALL
recvfrom(0x55,0x2a6c4000,0x1,0,0xbfbfb5f8,0xbfbfb694)
97625 httpd3.064293 GIO fd 85 read 30 bytes
97625 httpd3.064296 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd3.064299 RET recvfrom 30/0x1e
97625 httpd3.064308 CALL close(0x55)
I'm open to suggestions on where to look to improve this situation.
I'm not familiar with the kqueue event mechanism in BSD. I know it's
great, but I'm just not familiar with how to use it/etc.. If I'm
reading the syscall trace correctly, it looks like the daemon opens up a
socket to 127.0.0.1:53 (named) on fd 0x55/85, writes 30 bytes of data to
it, initiates a kernel event that writes 20 bytes of data to a different
descriptor 0x54/84, reads 20 bytes back from that fd, then reads 30
bytes from descriptor 0x55/85.
I wonder what the kevent() is actually accomplishing here. I wish I
could see within the kevent structs at 0xbfbfb678 and 0xbfbfb678, and
the timespec struct at 0xbfbfb68c.
There's also a part of me that remembers some sort of kevent/kqueue
problem on older FreeBSD that got fixed at one point. The problem is
that I can't remember what that problem was, nor what release fixed it.
As such I don't want to resort to a upgrade your OS! response.
Does this happen when httpd tries to do DNS resolution for, say, an
incoming connection to the web server (e.g. trying to resolve the
incoming IP address of the client to an FQDN), or is it happening within
some PHP code (assuming PHP is installed/used as an Apache module)
that's trying to do DNS resolution of some kind?
Are the delays always 3 seconds? If so, that almost sounds like a
timeout of some kind. That's the thing about the kevent() call: I wish
I could see what's in the timespec struct, since that's what defines the
timeout values.
--
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
On Sep 30, 2011, at 4:31 PM, Doug Barton wrote: o, this is a bit of an odd one I've got a web server running apache 2.2.17 and php 5.3.5. The host itself is running 7.3-RELEASE, i386, and is not busy. I can do DNS queries on the command line all day long and they are very snappy. Using nslookup, dig, whatever. Are you using prefork or worker/threaded MPM with Apache? While some PHP modules claim to be threadsafe, experience has left me convinced that neither threaded PHP nor threaded mod_perl is reliable under even minimal load. If you haven't tried using prefork MPM, consider using it, and maybe add fastcgi if you need to. The weirdness comes in when the httpd process needs to do a DNS lookup. [ ... ] I'm open to suggestions on where to look to improve this situation. One of the major problems with doing any DNS lookups in Apache is that you can easily encounter a DoS as all of the child processes try to resolve addresses; a malware scan coming from an IP with broken reverse DNS can cause things to grind to a halt for a few seconds. If at all possible, do not perform any DNS resolution in Apache, either for Allow/Deny rules in Location blocks, or for log processing. Regards, -- -Chuck ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: 7.3 + kqueue + apache/php + DNS lookup problem
Thanks Jeremy and Chuck. Answers below.
On 09/30/2011 17:37, Jeremy Chadwick wrote:
On Fri, Sep 30, 2011 at 04:31:18PM -0700, Doug Barton wrote:
Howdy,
So, this is a bit of an odd one I've got a web server running
apache 2.2.17 and php 5.3.5. The host itself is running 7.3-RELEASE,
i386, and is not busy. I can do DNS queries on the command line all day
long and they are very snappy. Using nslookup, dig, whatever.
The weirdness comes in when the httpd process needs to do a DNS lookup.
With a local cache I'm getting the following:
97625 httpd0.031139 CALL connect(0x55,0x284fd590,0x10)
97625 httpd0.031142 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd0.031150 RET connect 0
97625 httpd0.031153 CALL sendto(0x55,0x2a7d1000,0x1e,0,0,0)
97625 httpd0.031169 GIO fd 85 wrote 30 bytes
97625 httpd0.031173 RET sendto 30/0x1e
97625 httpd0.031179 CALL clock_gettime(0,0xbfbfb58c)
97625 httpd0.031184 RET clock_gettime 0
97625 httpd0.031188 CALL
kevent(0x54,0xbfbfb678,0x1,0xbfbfb678,0x1,0xbfbfb68c)
97625 httpd3.064266 GIO fd 84 wrote 20 bytes
note the 3 second delay here.
97625 httpd3.064277 GIO fd 84 read 20 bytes
97625 httpd3.064281 RET kevent 1
97625 httpd3.064287 CALL
recvfrom(0x55,0x2a6c4000,0x1,0,0xbfbfb5f8,0xbfbfb694)
97625 httpd3.064293 GIO fd 85 read 30 bytes
97625 httpd3.064296 STRU struct sockaddr { AF_INET, 127.0.0.1:53 }
97625 httpd3.064299 RET recvfrom 30/0x1e
97625 httpd3.064308 CALL close(0x55)
I'm open to suggestions on where to look to improve this situation.
I'm not familiar with the kqueue event mechanism in BSD. I know it's
great, but I'm just not familiar with how to use it/etc.. If I'm
reading the syscall trace correctly, it looks like the daemon opens up a
socket to 127.0.0.1:53 (named) on fd 0x55/85, writes 30 bytes of data to
it, initiates a kernel event that writes 20 bytes of data to a different
descriptor 0x54/84, reads 20 bytes back from that fd, then reads 30
bytes from descriptor 0x55/85.
I wonder what the kevent() is actually accomplishing here. I wish I
could see within the kevent structs at 0xbfbfb678 and 0xbfbfb678, and
the timespec struct at 0xbfbfb68c.
There's also a part of me that remembers some sort of kevent/kqueue
problem on older FreeBSD that got fixed at one point. The problem is
that I can't remember what that problem was, nor what release fixed it.
As such I don't want to resort to a upgrade your OS! response.
That's not necessarily off the table, but doing that would have to be
because we're sure it would fix the problem.
Does this happen when httpd tries to do DNS resolution for, say, an
incoming connection to the web server (e.g. trying to resolve the
incoming IP address of the client to an FQDN), or is it happening within
some PHP code (assuming PHP is installed/used as an Apache module)
that's trying to do DNS resolution of some kind?
It's a php module doing a lookup for the hostname of the back-end mysql
server.
Are the delays always 3 seconds?
Pretty much.
If so, that almost sounds like a timeout of some kind.
That was my first thought, but the answer always comes eventually.
To answer Chuck's questions, no threading is involved, and it's not
apache doing the lookups.
Doug
--
Nothin' ever doesn't change, but nothin' changes much.
-- OK Go
Breadth of IT experience, and depth of knowledge in the DNS.
Yours for the right price. :) http://SupersetSolutions.com/
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
