Control: tags -1 patch
Hi,
Thank you Vlad for the nice hint here!
I was searching for such curl option one year ago in Debian stretch
but it was missing there as curl there was only 7.52.
Sure, the fix is much nicer in Debian buster with new curl nowadays.
See attached fix #2 for approx behind HTTPS proxy which
is using "HTTP CONNECT" method.
Please consider to include it into default approx codebase.
P.S. the patch has been taken from GitHub:
> https://github.com/raol/approx-clone/commit/b5b2fb03d35bbc49ccf39661e024a5e2136daa00
P.P.S. I have also attached the Dockerfile for the test environment
where you can reproduce the issue. It contains stock approx and basic
squid proxy. Please see the steps to reproduce at the end of Dockerfile.
P.P.P.S If someone cannot wait for the fix here, you can
workaround the problem adding to /etc/approx/approx.conf:
> $curl_path /usr/bin/curl${IFS}--suppress-connect-headers
It will work for curl 7.54+ only (Debian buster+).
Thank you!
--
Alex Lutay
Head of Quality Assurance
Sipwise GmbH, Campus 21/Europaring F15
AT-2345 Brunn am Gebirge
FROM debian:buster
MAINTAINER Alex Lutay <alu...@sipwise.com>
RUN apt-get update && \
apt-get -y install approx curl vim less openbsd-inetd squid3 procps
RUN echo "9999 stream tcp nowait approx /usr/sbin/tcpd /usr/bin/env
http_proxy= https_proxy= /usr/sbin/approx" > /etc/inetd.conf && \
echo "debian-http http://debian.sipwise.com/debian"; >>
/etc/approx/approx.conf && \
echo "debian-https https://debian.sipwise.com/debian"; >>
/etc/approx/approx.conf && \
echo '$interval 1' >> /etc/approx/approx.conf && \
echo '$verbose true' >> /etc/approx/approx.conf && \
echo '$debug true' >> /etc/approx/approx.conf && \
\
sed -i 's/^#http_access allow localnet/http_access allow localnet/'
/etc/squid/squid.conf && \
echo "http_port 3129" >> /etc/squid/squid.conf && \
echo "acl sipwise dstdomain debian.sipwise.com" >> /etc/squid/squid.conf &&
\
echo "always_direct allow sipwise" >> /etc/squid/squid.conf
WORKDIR /root/
################################################################################
# Instructions for usage
# ----------------------
# When you want to build the base image from scratch:
# % docker build --no-cache --tag="approx-buster" .
# % docker run --rm -i -t approx-buster:latest bash
#
# Inside docker:
#
# 1) run: service rsyslog start # approx logs now in /var/log/syslog
# run: service openbsd-inetd start # now approx cache service is listening
on port 9999
# run: service squid start # start local proxy
#
# 2) run: curl
http://127.0.0.1:9999/debian-http/dists/buster/main/binary-amd64/Release
#
# The file 'Release' with proper content has been created in cache:
# > root@e7e3277d030a:~# cat
/var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release
# > Component: main
# > Origin: Debian
# > Architecture: amd64
# > Description: Debian buster mirror
# > root@e7e3277d030a:~#
#
# All OK here. This is an expected behaviour, approx has cached the file
properly.
# P.S The original URL is
http://debian.sipwise.com/debian/dists/buster/main/binary-amd64/Release
#
#
# 3) Let's test approx behind HTTP proxy first
#
# - change: sed -i "s#http_proxy= #http_proxy=http://127.0.0.1:3128/ #"
/etc/inetd.conf
# - restart: service openbsd-inetd restart
# - remove: rm
/var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release
# NOTE: the files will be returned from cache if available, remove it to
request new one from proxy!
#
# - test: curl
http://127.0.0.1:9999/debian-http/dists/buster/main/binary-amd64/Release
# - check: cat
/var/cache/approx/debian-http/dists/buster/main/binary-amd64/Release
# > Component: main
# > Origin: Debian
# > Architecture: amd64
# > Description: Debian buster mirror
# > root@3aab8dda86e2:~# # ALL OK here too, you can repeate curl command if
necessary
#
# JFYI squid logs in /var/log/squid/access.log:
# > 127.0.0.1 TCP_MISS/200 408 GET
http://debian.sipwise.com/debian/dists/buster/main/binary-amd64/Release -
HIER_DIRECT/192.168.208.45
#
#
# 4) Let's test approx behind HTTPS proxy
#
# - change: sed -i "s#https_proxy= #https_proxy=http://127.0.0.1:3129/ #"
/etc/inetd.conf
# - restart: service openbsd-inetd restart
# - remove: rm -f
/var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release
# NOTE: the files will be returned from cache if available, remove it to use
proxy!
#
# - test: curl
http://127.0.0.1:9999/debian-https/dists/buster/main/binary-amd64/Release
# NOTE: 1) curl connects http (NOT httpS) as approx doesn't support incoming
httpS
# 2) "debian-https" is in use here, see approx.conf: "debian-httpS
httpS://debian.sipwise.com/debian"
#
# - my test: curl
http://127.0.0.1:9999/debian-https/dists/buster/main/binary-amd64/Release
# > HTTP/1.1 200 OK
# > Date: Thu, 31 Oct 2019 09:14:25 GMT
# > Server: Apache
# > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT
# > ETag: "55-575330ff277aa"
# > Accept-Ranges: bytes
# > Content-Length: 85
# >
# > Component: main
# > Origin: Debian
# > Architecture: amd64
# > Description: Debian buster mirror
# > root@e90450035928:~# !!! Wrong here, we see headers here! and they are also
in cached file:
#
# - check: cat
/var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release
# > root@e90450035928:~# cat
/var/cache/approx/debian-https/dists/buster/main/binary-amd64/Release
# > HTTP/1.1 200 OK
# > Date: Thu, 31 Oct 2019 09:14:25 GMT
# > Server: Apache
# > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT
# > ETag: "55-575330ff277aa"
# > Accept-Ranges: bytes
# > Content-Length: 85
# >
# > Component: main
# > Origin: Debian
# > Architecture: amd64
# > Description: Debian buster mirror
# > root@3aab8dda86e2:~# # !!! File Release has been DAMAGED in cache !!!
#
# Issue explanation:
#
# root@e636398d4101:~# https_proxy=http://127.0.0.1:3129 /usr/bin/curl
--dump-header /tmp/zzz.log --fail --silent --header "Pragma: no-cache"
--include --header "If-Modified-Since: Thu, 30 Nov 2019 11:24:30 GMT"
"https://deb.sipwise.com/debian/dists/buster/main/binary-amd64/Release";
# > HTTP/1.1 200 Connection established
# >
# > HTTP/1.1 200 OK
# > Date: Thu, 31 Oct 2019 12:38:27 GMT
# > Server: Apache
# > Last-Modified: Thu, 06 Sep 2018 12:30:54 GMT
# > ETag: "55-575330ff277aa"
# > Accept-Ranges: bytes
# > Content-Length: 85
# >
# > Component: main
# > Origin: Debian
# > Architecture: amd64
# > Description: Debian buster mirror
# > root@e636398d4101:~#
#
# The first "200 Connection established" is "connected to proxy", the second
"200 OK" is connected to "debian.sipwise.com"
# approx removed the first section of headers but left the second one.
################################################################################
From b5b2fb03d35bbc49ccf39661e024a5e2136daa00 Mon Sep 17 00:00:00 2001
From: Oleg Rakitskiy <oleg.rakits...@gmail.com>
Date: Wed, 13 Nov 2019 21:05:46 +0200
Subject: [PATCH] Fix connection established response
small fix of the issue described here
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914798
new curl version starting from 7.54 supports suppressing
connect header, so used this options instead of
creating own workaround
---
url.ml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/url.ml b/url.ml
index ac185f2..99e73f2 100644
--- a/url.ml
+++ b/url.ml
@@ -56,7 +56,7 @@ let rate_option =
let curl_command options url =
Printf.sprintf
- "%s --fail --silent --header \"Pragma: no-cache\" %s %s %s"
+ "%s --fail --silent --header \"Pragma: no-cache\" --suppress-connect-headers %s %s %s"
curl_path rate_option (String.concat " " options) (quoted_string url)
let head_command = curl_command ["--head"]
--
2.1.4
