On 7/04/2013 6:45 a.m., Christopher H. Laco wrote:
I'm currently in the process of testing some software installs behind
a proxy and ran into something I don't quite understand.
While running behind a Squid proxy, apt-key calls were failing to
process key requests. The same requests run fine directly connected to
the internet.
From a machine directly connected to the internet:
/usr/bin/apt-key adv --keyserver hkp://keyserver.ubuntu.com:80
--recv-keys 765C5E49F87CBDE0
Executing: gpg --ignore-time-conflict --no-options
--no-default-keyring --secret-keyring /tmp/tmp.5JakWGN817
--trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg
--primary-keyring /etc/apt/trusted.gpg --keyserver
hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0
gpg: requesting key F87CBDE0 from hkp server keyserver.ubuntu.com
gpg: key F87CBDE0: "RCB Build <rcb-dep...@lists.rackspace.com>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
From a machine with proxy only access, http_proxy set in
/etc/environment & sudoers env_keep:
/usr/bin/apt-key adv --keyserver hkp://keyserver.ubuntu.com:80
--recv-keys 765C5E49F87CBDE0
Executing: gpg --ignore-time-conflict --no-options
--no-default-keyring --secret-keyring /tmp/tmp.9YzGDcyc3K
--trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg
--primary-keyring /etc/apt/trusted.gpg --keyserver
hkp://keyserver.ubuntu.com:80 --recv-keys 765C5E49F87CBDE0
gpg: requesting key F87CBDE0 from hkp server keyserver.ubuntu.com
gpgkeys: key 765C5E49F87CBDE0 not found on keyserver
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
And in the access.log on the proxy:
1365271816.885 294 10.10.10.20 TCP_MISS/403 3827 GET
http://keyserver.ubuntu.com/pks/lookup?op=get&options=mr&search=0xF87CBDE0
- DIRECT/91.189.89.49 text/html
I've further distilled this down to a simple curl statement against
keyserver.ubuntu.com, removing apt-key, gpg from the picture:
curl -v http://keyserver.ubuntu.com
...output here is the squid access denied error message page....
To be clear, at this point, this same server behind squid CAN access
other resources on the internet:
curl -v ubuntu.com
* About to connect() to proxy 10.10.10.10 port 3128 (#0)* Trying
10.10.10.10... connected
GET HTTP://ubuntu.com HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: ubuntu.com
Accept: */*> Proxy-Connection: Keep-Alive> * HTTP 1.0, assume close after body<
HTTP/1.0 301 Moved Permanently
< Date: Sat, 06 Apr 2013 18:41:26 GMT
< Server: Apache/2.2.14 (Ubuntu)
< Location: http://www.ubuntu.com/
< Content-Length: 306
< Content-Type: text/html; charset=iso-8859-1
< X-Cache-Lookup: MISS from localhost:3128
< Via: 1.0 localhost (squid/3.1.19)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
...output here is the rest of the / page...
And in the proxy access.log:
1365272030.197 161 10.10.10.20 TCP_MISS/403 3814 GET
http://keyserver.ubuntu.com/ - DIRECT/91.189.90.55 text/html
Now, on a machine with direct access to the internet:
curl -v http://keyserver.ubuntu.com
* About to connect() to keyserver.ubuntu.com port 80 (#0)
* Trying 91.189.89.49... connected
GET / HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: keyserver.ubuntu.com
Accept: */*
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Sat, 06 Apr 2013 18:16:49 GMT
< Server: sks_www/1.1.4
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: 0
< Content-Length: 1417
< Content-Type: text/html; charset=UTF-8
< X-Cache: MISS from localhost
< X-Cache-Lookup: MISS from localhost:11371
< Via: 1.0 localhost (squid/3.1.19)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
...remaining content of the actual / page on the keyserver...
Just for fun, if connect through a local TinyProxy instead of Squid, it works:
curl -v http://keyserver.ubuntu.com
* About to connect() to proxy 10.10.10.10 port 8888 (#0)* Trying
10.10.10.10... connected
GET http://keyserver.ubuntu.com HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: keyserver.ubuntu.com
Accept: */*> Proxy-Connection: Keep-Alive> * HTTP 1.0, assume close after body<
HTTP/1.0 200 OK
< Via: 1.0 localhost (squid/3.1.19), 1.1 tinyproxy (tinyproxy/1.8.3)
< Content-Length: 1417
< Expires: 0
< Server: sks_www/1.1.4
< Date: Sat, 06 Apr 2013 18:33:46 GMT
< X-Cache-Lookup: MISS from localhost:11371
< X-Cache: MISS from localhost
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=UTF-8
...remaining content of the actual / page on the keyserver...
I'm not an http/proxy guru. What I think is happening is that the
local Squid proxy receives the X-Cache MISS from upstream, and simply
halts the response as if it were a peer.
You say that BUT... the entire information about the failure consists of:
"
curl -vhttp://keyserver.ubuntu.com
...output here is the squid access denied error message page...
"
All the header traces I can see in your report are about the alternative tests
and working lookups. Nothing else about the important failure trace.
And no Squid does not make use of X-Cache HIT/MISS information. That is
purely for human debugging.
Can someone shed some insight on this? I always thought that a reverse
proxy should never expose it's X-Cache headers (probably for this very
reason).
Yes. No proxy should be sending the X-Cache headers unless some human is
debugging it. They were a debugging aid for PURGE and Digest requests
that go left in years back and nobody has got around to removing them
from all the response traffic where they are irrelevant. FWIW, those
headers will disappear sometime in the next couple of releases as we
improve the bandwidth usage and drop unnecessary HTTP/1.0 cruft.
Is this a local Squid configuration issue?
Is this a keyserver.ubuntu.com reverse proxy config issue?
Is this something else?
Will need the heades from that failing lookup to be sure.
Amos
