[squid-users] help needed to debug squid in daemon mode

[Ming-Ching Tiew]

I have a configuration where if I start squid with -N, it works. But if I run 
it without that, I will get child started, child exited a few times and 
eventually the parent process will die too. Because there is nothing in between 
the 'started' and 'exited' of the child process, I have no clues as to why the 
child exited. Short of modifying the source to do more printing, is that 
another better way to find out what's wrong ? 

Appreciate your comments.

[squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

I have set up a bridge according to instruction here :-

http://wiki.squid-cache.org/Features/Tproxy4

with squid 3.1.19 and kernel 3.2.21.

The configuration is working with other with most of the sites, except for 
yahoo mail. It's is extremely slow with yahoo mail, can hardly able to login 
and logout of yahoo mai. However the same computer when switch to nat REDIRECT 
using squid intercept, it is working OK, ie it is fast enough. 

Anyone observed the same issue ? 

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

[re-send - previous post might be deleted due to attachment ]

Attached please find the 'squid -X -N -d2 2>&1' ouput log when connecting to 
yahoo mail. When connecting to http://mail.yahoo.com, I get a 'No object data 
received'. When connecting to https, the bridge is not setup to intercept 
https, yet the login is hard to succeed.

Note that the same box configured to nat mode, it could interact with yahoo 
mail.

squid log --
2012/07/01 20:08:50.410| The request GET http://mail.yahoo.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:08:50.416| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:08:50.416| The request GET http://mail.yahoo.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:08:54.899| connReadWasError: FD 9: got flag -1
2012/07/01 20:08:54.904| ConnStateData::swanSong: FD 9
2012/07/01 20:08:55.490| The request GET 
http://us.mc1614.mail.yahoo.com/mc/welcome?.gx=1&.tm=1341173241&.rand=foctjdei2njpi
 is ALLOWED, because it matched 'localnet'
2012/07/01 20:08:55.501| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:08:55.501| The request GET 
http://us.mc1614.mail.yahoo.com/mc/welcome?.gx=1&.tm=1341173241&.rand=foctjdei2njpi
 is ALLOWED, because it matched 'localnet'
2012/07/01 20:10:10.020| connReadWasError: FD 9: got flag -1
2012/07/01 20:10:10.027| ConnStateData::swanSong: FD 9
2012/07/01 20:10:10.072| connReadWasError: FD 20: got flag -1
2012/07/01 20:10:10.079| connReadWasError: FD 18: got flag -1
2012/07/01 20:10:10.080| connReadWasError: FD 22: got flag -1
2012/07/01 20:10:10.080| connReadWasError: FD 10: got flag -1
2012/07/01 20:10:10.081| ConnStateData::swanSong: FD 20
2012/07/01 20:10:10.082| ConnStateData::swanSong: FD 18
2012/07/01 20:10:10.082| ConnStateData::swanSong: FD 22
2012/07/01 20:10:10.084| ConnStateData::swanSong: FD 10
2012/07/01 20:10:10.096| The request GET http://mail.yahoo.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:10:10.096| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:10:10.096| The request GET http://mail.yahoo.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:10:16.984| ctx: exit level  0
2012/07/01 20:10:16.992| WARNING: HTTP: Invalid Response: No object data 
received for http://mail.yahoo.com/ AKA mail.yahoo.com/
2012/07/01 20:10:16.994| fwdServerClosed: FD 10 http://mail.yahoo.com/
2012/07/01 20:10:23.515| ctx: exit level  0
2012/07/01 20:10:23.521| WARNING: HTTP: Invalid Response: No object data 
received for http://mail.yahoo.com/ AKA mail.yahoo.com/
2012/07/01 20:10:23.523| fwdServerClosed: FD 10 http://mail.yahoo.com/
2012/07/01 20:10:29.892| ctx: exit level  0
2012/07/01 20:10:29.899| WARNING: HTTP: Invalid Response: No object data 
received for http://mail.yahoo.com/ AKA mail.yahoo.com/
2012/07/01 20:10:29.901| fwdServerClosed: FD 10 http://mail.yahoo.com/
2012/07/01 20:10:29.915| The reply for GET http://mail.yahoo.com/ is ALLOWED, 
because it matched 'all'
--------

--- On Thu, 6/28/12, Ming-Ching Tiew  wrote:

> From: Ming-Ching Tiew 
> Subject: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 
> 3.2.21)
> To: squid-users@squid-cache.org
> Date: Thursday, June 28, 2012, 8:18 AM
> 
> I have set up a bridge according to instruction here :-
> 
> http://wiki.squid-cache.org/Features/Tproxy4
> 
> with squid 3.1.19 and kernel 3.2.21.
> 
> The configuration is working with other with most of the
> sites, except for yahoo mail. It's is extremely slow with
> yahoo mail, can hardly able to login and logout of yahoo
> mai. However the same computer when switch to nat REDIRECT
> using squid intercept, it is working OK, ie it is fast
> enough. 
> 
> Anyone observed the same issue ? 
> 

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

--- On Mon, 7/2/12, Ming-Ching Tiew  wrote:

> 
> Attached please find the 'squid -X -N -d2 2>&1' ouput
> log when connecting to yahoo mail. When connecting to http://mail.yahoo.com, 
> I get a 'No object data
> received'. When connecting to https, the bridge is not setup
> to intercept https, yet the login is hard to succeed.
> 
> Note that the same box configured to nat mode, it could
> interact with yahoo mail.
> 
> 

When connecting to https, squid won't see the https traffic as the bridge is 
not configured to tproxy the SSL traffic, but nevertheless, login is hard to 
succced and below is some of the http traffic while doing https yahoo mail 
login :-

2012/07/01 20:12:43.703| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:43.709| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:43.709| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:43.860| The reply for POST http://ocsp.digicert.com/ is 
ALLOWED, because it matched 'all'
2012/07/01 20:12:43.897| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:43.897| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:43.897| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:43.999| The reply for POST http://ocsp.digicert.com/ is 
ALLOWED, because it matched 'all'
2012/07/01 20:12:45.897| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:45.902| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:45.902| The request POST http://ocsp.digicert.com/ is ALLOWED, 
because it matched 'localnet'
2012/07/01 20:12:45.994| The reply for POST http://ocsp.digicert.com/ is 
ALLOWED, because it matched 'all'
2012/07/01 20:12:56.429| The request GET 
http://us.mc1614.mail.yahoo.com/mc/welcome?.gx=1&.tm=1341173481&.rand=82c3g22q15e9c
 is ALLOWED, because it matched 'localnet'
2012/07/01 20:12:56.434| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:56.434| The request GET 
http://us.mc1614.mail.yahoo.com/mc/welcome?.gx=1&.tm=1341173481&.rand=82c3g22q15e9c
 is ALLOWED, because it matched 'localnet'
2012/07/01 20:12:57.172| The reply for GET 
http://us.mc1614.mail.yahoo.com/mc/welcome?.gx=1&.tm=1341173481&.rand=82c3g22q15e9c
 is ALLOWED, because it matched 'all'
2012/07/01 20:12:58.223| ConnStateData::swanSong: FD 10
2012/07/01 20:12:58.288| The request GET 
http://ads.bluelithium.com/pixel?id=365083&t=2 is ALLOWED, because it matched 
'localnet'
2012/07/01 20:12:58.293| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:58.293| The request GET 
http://ads.bluelithium.com/pixel?id=365083&t=2 is ALLOWED, because it matched 
'localnet'
2012/07/01 20:12:58.340| The request GET
 
http://us.bc.yahoo.com/b?P=jdldrmKL1cIRitGnT.7kIgfHr465vE_wrusADSBa&T=19th47hua%2fX%3d1341173483%2fE%3d398301041%2fR%3dmail%2fK%3d5%2fV%3d1.1%2fW%3dJ%2fY%3dYAHOO%2fF%3d4183513604%2fH%3dY29udGVudD0ibm9fZXhwYW5kYWJsZTthamF4X2NlcnRfZXhwYW5kYWJsZTsiIHNlcnZlSWQ9ImpkbGRybUtMMWNJUml0R25ULjdrSWdmSHI0NjV2RV93cnVzQURTQmEiIHNpdGVJZD0iNDQ1NDU1MSIgdFN0bXA9IjEzNDExNzM0ODMwMzIwNzkiIA--%2fI%3d1%2fS%3d1%2fJ%3d23D48B62&U=13f0b5vp5%2fN%3dDNsnCWKL5No-%2fC%3d289534.10180982.10848075.9860700%2fD%3dFOOT%2fB%3d4386606%2fV%3d1&U=13fp61862%2fN%3dGdsnCWKL5No-%2fC%3d624324.13382210.13481662.12549985%2fD%3dSIP%2fB%3d5677395%2fV%3d1&U=12dr6v6qs%2fN%3dG9snCWKL5No-%2fC%3d-1%2fD%3dFSRVY%2fB%3d-1%2fV%3d0&U=12b0qkkrn%2fN%3dHNsnCWKL5No-%2fC%3d-1%2fD%3dRMP%2fB%3d-1%2fV%3d0&U=12a2tolqt%2fN%3dE9snCWKL5No-%2fC%3d-1%2fD%3dN2%2fB%3d-1%2fV%3d0&U=12bacet7j%2fN%3dEtsnCWKL5No-%2fC%3d-1%2fD%3dMNW%2fB%3d-1%2fV%3d0&U=12a7d0brd%2fN%3dFtsnCWKL5No-%2fC%3d-1%2fD%3dRS%2fB%3d-1%2fV%3d0&U=12bar1525%2fN%3dF9
snCWKL5No-%2fC%3d-1%2fD%3dRS2%2fB%3d-1%2fV%3d0&Q=0&O=0.43259778993149667 is 
ALLOWED, because it matched 'localnet'
2012/07/01 20:12:58.340| client_side_request.cc(556) clientAccessCheck2: No 
adapted_http_access configuration.
2012/07/01 20:12:58.340| The request GET
 
http://us.bc.yahoo.com/b?P=jdldrmKL1cIRitGnT.7kIgfHr465vE_wrusADSBa&T=19th47hua%2fX%3d1341173483%2fE%3d398301041%2fR%3dmail%2fK%3d5%2fV%3d1.1%2fW%3dJ%2fY%3dYAHOO%2fF%3d4183513604%2fH%3dY29udGVudD0ibm9fZXhwYW5kYWJsZTthamF4X2NlcnRfZXhwYW5kYWJsZTsiIHNlcnZlSWQ9ImpkbGRybUtMMWNJUml0R25ULjdrSWdmSHI0NjV2RV93cnVzQURTQmEiIHNpdGVJZD0iNDQ1NDU1MSIgdFN0bXA9IjEzNDExNzM0ODMwMzIwNzkiIA--%2fI%3d

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

--- On Mon, 7/2/12, Eliezer Croitoru  wrote:


> it works slowly for all clients or just windows 7 ? other
> clients?
> i have seen a problem when applying tproxy on a router with
> win7 client.
> from unknown reason using standard routing and intercept
> everything 
> works fine but when i switched to tproxy all http access
> from this win7 
> machine was slow as hell until i restarted the machine.
> then everything works fine.
> on the same time i had a linux client on the setup that
> worked without 
> any problem.
> 
> if you are having the same symptom i think it's a windows
> problem.
> 
> Regards,
> Eliezer
> 
> 

No your symptom and mine are totally different. With the limited testing, I 
don't see any problem with any OSes, any sites. I only see problem when 
visiting yahoo mail. Meaning when I use Windows XP, firefox, IE, Linux with 
firefox to visit any sites, the response is decent and acceptable. I just can't 
go to yahoo mail (classic). But when I switch to nat REDIRECT mode, it works 
perfectedly including yahoo mail.

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

--- On Mon, 7/2/12, Ming-Ching Tiew  wrote:


> 
> No your symptom and mine are totally different. With the
> limited testing, I don't see any problem with any OSes, any
> sites. I only see problem when visiting yahoo mail. Meaning
> when I use Windows XP, firefox, IE, Linux with firefox to
> visit any sites, the response is decent and acceptable. I
> just can't go to yahoo mail (classic). But when I switch to
> nat REDIRECT mode, it works perfectedly including yahoo
> mail.
> 
> 
> 

I tried with kernel 2.6.37.6 and had the same problem. Further to it, also 
encounter another website which will hit the same problem. So it must be either 
the squid version or my setup is wrong somewhere. But the strange thing is 
those web sites which work, continue to work while those websites (such as 
yahoo mail) continue to give the same problem. Appreciate any comments about 
the next best thing to try. 

Thanks.

Re: [squid-users] Re: transparent (intercepting?) without wccp, options?

[Ming-Ching Tiew]

--- On Fri, 7/6/12, Ezequiel Birman  wrote:


> 
> In http://wiki.squid-cache.org/Features/Tproxy4#Routing_configuration
> I
> see rules applied to eth0, should i rewrite br0 in place of
> eth0?
> 
> 

I think it should rather be lo. 

I did not follow the rp_filter thingie strictly. I set all of them zero.

Re: [squid-users] Re: transparent (intercepting?) without wccp, options?

[Ming-Ching Tiew]

- Original Message -

>for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
>echo 2 > $i
>done

Really strange. I have never able to get tproxy to work unless I switch the 
rp_filter to 0. 

When rp_filter is 2, I could sniff the traffic, but somehow squid is not able 
to see it. 

Re: [squid-users] i'm having a little performance trouble with squid + ICAP server.

[Ming-Ching Tiew]

Sorry I am offering no help but I am interested to know how do you set up a 
stress test environment.
I supposed it's an automatic script based stress tests ?

Re: [squid-users] Squis 3.1.20 doest not compile with eCAP 0.2.0

[Ming-Ching Tiew]

squid 3.1.20 is supposed to be compiled with eCap 0.0.3, not ecap 0.2.0.
squid 3.2 can be compiled with ecap 0.2.0.

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

When logging out from yahoo mail, it's very slow and eventually there is any 
error.

Don't get that when configured to use nat mode.
<>

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

I will setup a new machine and report back. It's will be fedora 15, i386 

because that's the latest DVD I have. Need be I will recompile a newer kernel.



- Original Message -
From: Eliezer Croitoru 
To: squid-users@squid-cache.org
Cc: 
Sent: Wednesday, July 18, 2012 8:08 PM
Subject: Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 
3.2.21)

On 7/18/2012 11:35 AM, Felix Leimbach wrote:
> Hi,
> 
> On 07/18/2012 04:28 AM, Ming-Ching Tiew wrote:
>> When logging out from yahoo mail, it's very slow and eventually there
>> is any error.
> 
> I'm not sure whether this is your problem - but I too had similar
> problems with 3.1.19.
> Upgrading to 3.1.20 solved the problem - turned out bug 3466 ("adaption
> stuck on last single-byte body piece") was the culprit.
> Try giving 3.1.20 a shot.
> 
> HTH
> Felix
by the screenshot he is using 3.1.20.
well i do not get this problem with wither squid 3.1.16-20 or 3.2.0-8-17
so it can be a network issue (other proxy in the way\routing etc) or develop 
libs dependency.
from his logs before:
2012/07/01 20:10:16.992| WARNING: HTTP: Invalid Response: No object data 
received for http://mail.yahoo.com/ AKA mail.yahoo.com/
2012/07/01 20:10:16.994| fwdServerClosed: FD 10 http://mail.yahoo.com/
if he is getting the problem i would like to make effort reproduce it.

so more data needed:
OS = linux
32 \ 64 bit = ?
what Distribution ?
uname -a output ?
what are the configure options for squid ? (squid -v output)
if a package has being used which? (download source).
tproxy as router?
do you intercept ssl?


any data will give more info on the problem.

tcpdump -i any 'port 80' -n
output while the problem accrues is will be very good.

iptables-save
ip route
ip rule


some more data will be helpful instead of just throwing to the air the problem 
with the log declaring about the problem.

as for http://mail.yahoo.com/
this is a 302 "HTTP/1.0 302 Moved Temporarily" reply so it might be something 
with the size of the reply.

try to run
curl  -v  http://mail.yahoo.com/
to see if you get any output while not using squid.

Eliezer

-- Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer  ngtech.co.il

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

OK I could see the same problem with just fedora 15. The client side I use a 
Window XP 
machine loaded with Firefox and Internet Explorer. What I experience with this 
set up
is that, it is impossible to logon on to Yahoo mail using Firefox. But in other 
occasions,
from home internet, I have experienced before, it is impossible to logoff/logon 
and slow 

from yahoo mail using Internet Explorer. But when I switch to intercepting 
mode, it's
speedy.

This is what I did :-

1. Install fedora 15
2. Disable Fedora 15's renaming of network interface.
  because I would to like use back eth0, eth1.
3. yum install bridge-utils and ebtables
4. Delete NetworkManager
5. Disable SElinux.
6. yum install squid, fedora 15 is using squid-3.1.19.
    Configure /etc/squid.conf for tproxy, basically just added 'http_port 3129 
tproxy'.
7. Modify /etc/rc.local to perform all the setup of bridge, iptables and 
routing.
    I have attached my rc.local. 

8. After the computer fully booted, then I manually start up squid, I would do
 squid -N -X -d2 or squid -sY.


rc.local attached.

iptables inline below
# Generated by iptables-save v1.4.10 on Sat Jul 21 07:29:03 2012
*nat
:PREROUTING ACCEPT [17:991]
:INPUT ACCEPT [17:991]
:OUTPUT ACCEPT [81:4793]
:POSTROUTING ACCEPT [81:4793]
COMMIT
# Completed on Sat Jul 21 07:29:03 2012
# Generated by iptables-save v1.4.10 on Sat Jul 21 07:29:03 2012
*mangle
:PREROUTING ACCEPT [201:17028]
:INPUT ACCEPT [278:26348]
:FORWARD ACCEPT [128:7680]
:OUTPUT ACCEPT [187:31351]
:POSTROUTING ACCEPT [325:40825]
-A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port 3129 --on-ip 0.0.0.0 
--tproxy-mark 0x1/0x1 
COMMIT
# Completed on Sat Jul 21 07:29:03 2012
# Generated by iptables-save v1.4.10 on Sat Jul 21 07:29:03 2012
*filter
:INPUT ACCEPT [30:2749]
:FORWARD ACCEPT [128:7680]
:OUTPUT ACCEPT [186:31171]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
COMMIT
# Completed on Sat Jul 21 07:29:03 2012



- Original Message -
From: Eliezer Croitoru 

so more data needed:
OS = linux
32 \ 64 bit = ?
what Distribution ?
uname -a output ?
what are the configure options for squid ? (squid -v output)
if a package has being used which? (download source).
tproxy as router?
do you intercept ssl?


any data will give more info on the problem.

tcpdump -i any 'port 80' -n
output while the problem accrues is will be very good.

iptables-save
ip route
ip rule


some more data will be helpful instead of just throwing to the air the problem 
with the log declaring about the problem.

as for http://mail.yahoo.com/
this is a 302 "HTTP/1.0 302 Moved Temporarily" reply so it might be something 
with the size of the reply.

try to run
curl  -v  http://mail.yahoo.com/
to see if you get any output while not using squid.

Eliezer

-- Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer  ngtech.co.il

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
>From: Ming-Ching Tiew 
>
> rc.local attached.
>


Attachment rejected so re-post inline below :-

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

ifconfig eth0 0.0.0.0 up
ifconfig eth1 0.0.0.0 up
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth1
ifconfig br0 192.168.1.253 up
ip route add default via 192.168.1.1


MODE=tproxy
if [ "$MODE" = "tproxy" ]
then
  ebtables -t broute -F BROUTING 
  ebtables -t broute -A BROUTING -i eth0 -p ipv4 --ip-protocol tcp \
  --ip-destination-port 80 -j redirect --redirect-target DROP
  ebtables -t broute -A BROUTING -i eth1 -p ipv4  --ip-protocol tcp \
  --ip-source-port 80 -j redirect --redirect-target DROP
  ebtables -t broute -A BROUTING -i eth1 -p ipv4 --ip-protocol tcp \
  --ip-destination-port 80 -j redirect --redirect-target DROP
  ebtables -t broute -A BROUTING -i eth0 -p ipv4  --ip-protocol tcp \
  --ip-source-port 80 -j redirect --redirect-target DROP
  cd /proc/sys/net/bridge
  for i in *
  do
   echo 0 > $i
  done

  iptables -t mangle -F
  iptables -t nat -F
  iptables -t mangle -F PREROUTING
  iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
  iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY \
   --tproxy-mark 0x1/0x1 --on-port 3129

  iptables -t mangle -A DIVERT -j MARK --set-mark 1
  iptables -t mangle -A DIVERT -j ACCEPT
  ip rule delete fwmark 1/1 lookup 150 2>/dev/null
  ip rule add fwmark 1/1 lookup 150
  ip route flush table 150
  ip route add local 0.0.0.0/0 dev lo table 150
  for i in /proc/sys/net/ipv4/conf/*/rp_filter
  do
    echo 0 > $i
  done
  sed -i -e 's/http_port 3129.*/http_port 3129 tproxy/' /etc/squid/squid.conf
else
  # intercepting nat MODE
  ebtables -t broute -F
  ebtables -t broute -A BROUTING -p ipv4 --ip-protocol tcp --ip-source-port 80 \
    -j redirect --redirect-target ACCEPT
  iptables -t nat -F 
  iptables -t mangle -F
  iptables -t nat -A PREROUTING -i br0 -p tcp --dport 80 -j REDIRECT --to-port 
3129 
  sed -i -e 's/http_port 3129.*/http_port 3129 intercept/' /etc/squid/squid.conf
fi
# Default Fedora DVD installation has rules which must be deleted
iptables -D INPUT   -j REJECT --reject-with icmp-host-prohibited
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited
echo 1 > /proc/sys/net/ipv4/ip_forward

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
> From: Eliezer Croitoru 
>
> so what you just need for ebtables is two rules:
> all packets the are destined to the web om port 80.. route them into the 
> machine... later will be intercepted by tproxy > so:
> ebtables -t broute -A BROUTING -i eth0 -p ipv4 --ip-protocol tcp \ 
> --ip-destination-port 80 -j redirect --redirect-target DROP

> and every packet that comes from the internet from port 80 (web server) 
> should be always get to the proxy as it's an > answer to squid request either 
> tproxy or intercept.
> the only difference with intercept mode is that:
> the packet that comes back from the internet destination is the proxy and on 
> any case the bridge will send it to the > proxy.

> so to intercept web answers to the proxy you need the rules:
> ebtables -t broute -A BROUTING -i eth1 -p ipv4  --ip-protocol tcp \
> --ip-source-port 80 -j redirect --redirect-target DROP
> 
> and that is it for the bridge.

Your rules are essentially the same as mine and I don't see how it that 
different,
maybe I am just missed the point.


The reason you see many more rules than is needed because I want to make them
the connection symmetric so that it does not matter which ethX is the upstream,
and which is the down stream, ie whichever port you plug into it will still 
work. 

And I have specifically confirmed that the other two additional rules have no 
traffic.

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

Thank you for the input. I will do that sometime later and report back
when I have new info.



- Original Message -
From: Eliezer Croitoru 

they indeed are not suppose to fail your setup but it's not suppose to 
be symmetric with tproxy.
the idea of the bridge is that you have clients side and external side 
that you abuse both.

if you make it this way for a purpose it's another story.
i would say that the result can show some really nasty issue you are 
having in the network level and ebtables+switch is the basic thing to check.
i will try to dump the tcp sessions on the interfaces using:
tcpdump -i any -X -s0 -n port 80 -w test.pcap

i will be happy to look into the packets to see if there is a clue in 
them saying something about the "zero reply".

to make sure it's not squid issue try to install the rpm of squid 3.2
http://pkgs.org/fedora-16/fedora-i386/squid-3.2.0.12-1.fc16.i686.rpm.html

i have tested it on fedora 15-16 and still the same result that it works 
both on 3.1.X and 3.2.X.

you can try to play with stp on\off on the bridge for case of packets 
getting lost somewhere by STP filters.

Regards,
Eliezer

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer  ngtech.co.il

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
From: Eliezer Croitoru 

> i would say that the result can show some really nasty issue you are 
> having in the network level and ebtables+switch is the basic thing to check.
> i will try to dump the tcp sessions on the interfaces using:
> tcpdump -i any -X -s0 -n port 80 -w test.pcap

> i will be happy to look into the packets to see if there is a clue in 
> them saying something about the "zero reply".

> to make sure it's not squid issue try to install the rpm of squid 3.2
> http://pkgs.org/fedora-16/fedora-i386/squid-3.2.0.12-1.fc16.i686.rpm.html

> i have tested it on fedora 15-16 and still the same result that it works 
> both on 3.1.X and 3.2.X.

On the same fedora 15 machine, I loaded the squid-3.2.0.12 based on the link 
above and that seems to work better based on the limited test. Login/logout
from the home internet seems OK. I looked at the configure options, and 
compile it myself the same program, and that seems to work too.

However, since what I have previously tested, one of them was squid-3.2.0.18,
and so I compile it using same configure options, and it did NOT work, ie my 
logon
to yahoo mail using internet explorer 8 hang until timeout and browser gets 
zero 
sized reply. Using firefox seems OK.

And I also have captured the test.pcap which I will send in a separate mail to 
you directly instead of mail list as it will be rejected by maillist.

The configure options are included inline below :-

./configure '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' \
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' \
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' \
'--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' \
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' \
'--libexecdir=/usr/lib/squid' '--localstatedir=/var' \
'--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' \
'--with-logdir=$(localstatedir)/log/squid' \
'--with-pidfile=$(localstatedir)/run/squid.pid' \
'--disable-dependency-tracking' '--enable-arp-acl' \
'--enable-follow-x-forwarded-for' '--enable-auth' \
'--enable-auth-basic=DB,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' 
 \
'--enable-auth-ntlm=smb_lm,fake' \
'--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos' \
'--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
 \
'--enable-cache-digests' '--enable-cachemgr-hostname=localhost' \
'--enable-delay-pools' '--enable-epoll' '--enable-icap-client' \
'--enable-ident-lookups' '--with-large-files' '--enable-linux-netfilter' \
'--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' \
'--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' \
'--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' \
'--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 


The test is very repeated, ie when I 'make install' from squid-3.2.0.12 it 
works but not
squid-3.2.018.

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
From: Ming-Ching Tiew 
To: "squid-users@squid-cache.org" 

> The test is very repeated, ie when I 'make install' from squid-3.2.0.12 it 
> works but not
> squid-3.2.018.

I meant the tests were very repeatable, squid-3.2.0.12 works, squid-3.2.0.13 
works.
Squid-3.2.0.14 onwards ( tested squid-3.2.0.14, squid-3.2.0.15, squid-3.2.0.16, 
squid-3.2.0.18 ) all start giving problems. 

For squid-3.2.0.14, when I try to logon to yahoo mail, I get this thing below. 
Other
versions seem to just hang until timeout. I am not trying to finger point at 
squid
beta version, but I hope these tests will throw some lights to my problem with
using squid in tproxy mode :-

ERROR
The requested URL could not be retrieved



Invalid Request error was encountered while trying to process the request:

GET /neo/launch?.rand=b1ktfi57od9dm HTTP/1.1
Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, 
application/x-shockwave-flash, application/vnd.ms-powerpoint, 
application/vnd.ms-excel, application/msword, application/x-ms-xbap, 
application/vnd.ms-xpsdocument, application/xaml+xml, 
application/x-ms-application, */*
Accept-Language: en-us,zh-CN;q=0.7,zh-TW;q=0.3
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 
1.6); .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)
Accept-Encoding: gzip, deflate
Host: us.mg6.mail.yahoo.com
Connection: Keep-Alive
Cookie: YM.NEO_401430109=; 
B=3c13u8l80rhtv&b=4&d=fhZ_4jJpYFpJGdj7zY3momBX59rysCEBsTfUSw--&s=4n&i=bKhuXma.oOGyPs48n8Zs;
 MSC=t=1343095009X; 
CH=AgBQDcfQACnOUAA0o0AANfIwOTMwUBU3zFAABaHQAAU60AARnaAAKebQAD78kBI1E8AFFzjABS/oMCMzW4AVKto=;
 BA=ba=954&ip=202.46.125.50&t=1343093816; 
F=a=T0kCqogMvSg2966rgvZtTBroHDD9YRIxrhruO5G2Id9YECK2VTCcJQC_osnCHco7IulOaBU-&b=M6y3;
 
Y=v=1&n=b76g5t5k0htpl&l=abvss/o&p=m1tvvmy313220400&iz=52200&r=3l&lg=en-US&intl=us;
 C=mg=1; 
U=mt=qIPNYp2MhYjyHJ68oBizJjAhX4ZPw4zZZ3fShTnx&ux=oufDQB&un=b76g5t5k0htpl; 
YLS=v=1&p=1&n=0; PH=fn=mDjSuQdea6k2x1s-&l=en-US&i=us; 
T=z=pEgDQBpYHIQBR9xcpMzkeu.NjE1NwYzNzYzNDc2N04-&a=QAE&sk=DAALMXwKvPWVxg&ks=EAARH9uRmqjEAc3_qrdxUidnQ--~E&d=c2wBTVRZeU1BRTBNREUwTXpBeE1Eay0BYQFRQUUBZwFLVjc0SFhBNkg2UkFSTFgyS0JUM01CM0cyWQF0aXABSzdTZnlBAXp6AXBFZ0RRQkE3RQ--;
 
RT=s=1343095221696&u=&r=https%3A//login.yahoo.com/config/login_verify2%3F%26.src%3Dym
Some possible problems are:

Missing or unknown request method.

Missing URL.

Missing HTTP Identifier (HTTP/1.0).

•Request is too large.

•Content-Length missing for POST or PUT requests.

•Illegal character in hostname; underscores are not allowed.

•HTTP/1.1 “Expect:” feature is being asked from an HTTP/1.0 software.

Your cache administrator is webmaster.






Generated Tue, 24 Jul 2012 02:00:18 GMT by fedora15 (squid/3.2.0.14)

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
From: Ming-Ching Tiew 
To: "squid-users@squid-cache.org" 

> The test is very repeated, ie when I 'make install' from squid-3.2.0.12 it 
> works but not
> squid-3.2.018.

I meant the tests were very repeatable, squid-3.2.0.12 works, squid-3.2.0.13 
works.
Squid-3.2.0.14 onwards ( tested squid-3.2.0.14, squid-3.2.0.15, squid-3.2.0.16, 
squid-3.2.0.18 ) all start giving problems. 

For squid-3.2.0.14, when I try to logon to yahoo mail, I get this thing below. 
Other
versions seem to just hang until timeout. I am not trying to finger point at 
squid
beta version, but I hope these tests will throw some lights to my problem with
using squid in tproxy mode :-

ERROR
The requested URL could not be retrieved



Invalid Request error was encountered while trying to process the request:

GET /neo/launch?.rand=b1ktfi57od9dm HTTP/1.1
Accept:
 image/gif, image/jpeg, image/pjpeg, image/pjpeg, 
application/x-shockwave-flash, application/vnd.ms-powerpoint, 
application/vnd.ms-excel, application/msword, application/x-ms-xbap, 
application/vnd.ms-xpsdocument, application/xaml+xml, 
application/x-ms-application, */*
Accept-Language: en-us,zh-CN;q=0.7,zh-TW;q=0.3
User-Agent:
 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 
1.6); .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)
Accept-Encoding: gzip, deflate
Host: us.mg6.mail.yahoo.com
Connection: Keep-Alive
Cookie:
 YM.NEO_401430109=; 
B=3c13u8l80rhtv&b=4&d=fhZ_4jJpYFpJGdj7zY3momBX59rysCEBsTfUSw--&s=4n&i=bKhuXma.oOGyPs48n8Zs;
 MSC=t=1343095009X; 
CH=AgBQDcfQACnOUAA0o0AANfIwOTMwUBU3zFAABaHQAAU60AARnaAAKebQAD78kBI1E8AFFzjABS/oMCMzW4AVKto=;
 BA=ba=954&ip=202.46.125.50&t=1343093816; 
F=a=T0kCqogMvSg2966rgvZtTBroHDD9YRIxrhruO5G2Id9YECK2VTCcJQC_osnCHco7IulOaBU-&b=M6y3;
 
Y=v=1&n=b76g5t5k0htpl&l=abvss/o&p=m1tvvmy313220400&iz=52200&r=3l&lg=en-US&intl=us;
 C=mg=1; 
U=mt=qIPNYp2MhYjyHJ68oBizJjAhX4ZPw4zZZ3fShTnx&ux=oufDQB&un=b76g5t5k0htpl;
 YLS=v=1&p=1&n=0; PH=fn=mDjSuQdea6k2x1s-&l=en-US&i=us; 
T=z=pEgDQBpYHIQBR9xcpMzkeu.NjE1NwYzNzYzNDc2N04-&a=QAE&sk=DAALMXwKvPWVxg&ks=EAARH9uRmqjEAc3_qrdxUidnQ--~E&d=c2wBTVRZeU1BRTBNREUwTXpBeE1Eay0BYQFRQUUBZwFLVjc0SFhBNkg2UkFSTFgyS0JUM01CM0cyWQF0aXABSzdTZnlBAXp6AXBFZ0RRQkE3RQ--;
 
RT=s=1343095221696&u=&r=https%3A//login.yahoo.com/config/login_verify2%3F%26.src%3Dym
Some possible problems are:

Missing or unknown request method.

Missing URL.

Missing HTTP Identifier (HTTP/1.0).

•Request is too large.

•Content-Length missing for POST or PUT requests.

•Illegal character in hostname; underscores are not allowed.

•HTTP/1.1 “Expect:” feature is being asked from an HTTP/1.0 software.

Your cache administrator is webmaster.






Generated Tue, 24 Jul 2012 02:00:18 GMT by fedora15 (squid/3.2.0.14)

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
From: Amos Jeffries 
To: squid-users@squid-cache.org

> One big change in 3.2.0.14 related to TPROXY traffic handling. A bug in 
> host_strict_verify was fixed, making the validation > bypass properly when 
> the (default) non-strict was configured.
>
> - check that this host_strict_verify directive is ABSENT from your config 
> file, or at very least set to OFF.

There is not such directive in my config file.

> 
> - check your cache.log for host forgery security alerts, or forwarding loop 
> warnings when these requests are being made.
>
> - check your cache.log file for invalid request parsing messages. This may 
> require "debug_options ALL,1" to be configured.

The cache.log has these :-

2012/07/24 12:38:34.628| SECURITY ALERT: Host header forgery detected on 
local=219.93.13.235:80 remote=192.168.1.3 FD 13 flags=17 (local IP does not 
match any domain IP)
2012/07/24 12:38:34.628| SECURITY ALERT: By user agent: Mozilla/4.0 
(compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; (R1 1.6); .NET CLR 
3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)
2012/07/24 12:38:34.628| SECURITY ALERT: on URL: 
http://us.mg6.mail.yahoo.com/neo/launch?.rand=5fsn8p9a1efna

What is the significance ? Is it that my test client machine is infected by 
virus adware or what ? 

Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)

[Ming-Ching Tiew]

- Original Message -
From: Amos Jeffries 
To: squid-users@squid-cache.org

> The HTTP Host: header contains a domain name which does not match the IP 
> address the TCP connection is being 

> made to. http://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery covers 
> the problem in some detail. For your 

> case in particular I suspect the DNS situations need to be checked.

> 219.93.13.235 found by the client is not one of the IPs belonging to 
> us.mg6.mail.yahoo.com which DNS is supplying
> to Squid. On the "big name" websites this is usually caused by Geo-DNS 
> resolution problems rather than client infection.
> But there is no way for Squid to be sure. The only option is for Squid to 
> open a TCP connection directly to that IP 

> and hope for the best, or if direct connections are blocked the unable to 
> connect comes up.
>
> NOTE: if you are using cache_peer you can currently only send them requests 
> where the Host header validates as okay,
> or were received as regular forward-proxy / reverse-proxy traffic. (I'm 
> working on that one as I type, but the fix is a few
> days/weeks away).
>
> If you are *not* using cache_peer then you have TCP connectivity problems 
> that need fixing.
> 
> You can run 3.1 series for now, or that older beta (ideally not, but if you 
> *really* have to its okay for now). There 

> are tweaks and improvements around this right up to the 
> squid-3.2.0.18-20120724-r11624 

>  snapshot 
> with more coming. With 

> probably some of the network environment situations mentioned in the wiki 
> needing to be fixed as well.
> 
> Amos

As it seems the header forgery is likely a sidetrack issue due to me using 
different
name servers in squid machine and test client machine. After I synchronized the 
name 

server to be the same, that message does not appear anymore. But still my 
problem 

of unable to logon to yahoo mail in tproxy mode using squid-3.2.0.14 is still 
there
( logon using intercepting mode is ok ), whereas when using squid-3.2.0.12 and 

3.2.0.13, I could logon to yahoo mail.

Therefore the "significant" changes in squid-3.2.0.14 might throw some lights 
as to
why I could not logon on to yahoo mail in tproxy mode.

[squid-users] tproxy can't connect to target url after url rewrite program to different host

[Ming-Ching Tiew]

Tested this with Squid Version 3.1.20-20120710-r10457,

After a simple url_rewrite_program changing from url to
a different host, the communication will not succeed 
( using linux bridge with ebtables/iptables for this tproxy 

communication ).

The nat intercept mode could succeed.

Re: [squid-users] tproxy can't connect to target url after url rewrite program to different host

[Ming-Ching Tiew]

From: Eliezer Croitoru 
To: squid-users@squid-cache.org
Cc: 
Sent: Saturday, July 28, 2012 10:53 AM
Subject: Re: [squid-users] tproxy can't connect to target url after url rewrite 
program to different host

On 07/28/2012 02:55 AM, Ming-Ching Tiew wrote:
>>
>> Tested this with Squid Version 3.1.20-20120710-r10457,
>>
>> After a simple url_rewrite_program changing from url to
>> a different host, the communication will not succeed
>> ( using linux bridge with ebtables/iptables for this tproxy
>>
>> communication ).
>>
>> The nat intercept mode could succeed.
>only for the url?
>for me it works fine.

Further testing revealed that if the re-written url is at port 80,
then it works. If the url is changed to a different port, then
it will timeout. Eg 


   http://dfsdffsa:8080/fsdafsdf

Looks like there is a restriction here, because when squid 

opens a connection faking the client  (tproxy), the reply since is it 

not port 80, it is not coming back to squid.

Re: [squid-users] tproxy can't connect to target url after url rewrite program to different host

[Ming-Ching Tiew]

From: Eliezer Croitoru 
To: squid-users@squid-cache.org

> now that you remind me.
> i have seen this kind of problem!!!
> it was nasty on squid 3.1.
> you can see in iptables connection tracking that squid is opening the 
> socket but it sends the first syn and wont get the incoming syn from the 
> destination.
>
> but there are two different situations bridge and routing.
> on bridge it's pretty obviates.
> you must tell the bridge to "drop" the incoming traffic from of source 
> port 8080 otherwise it will be bridged to the client and wont get back 
> to squid.
>


If it is an external web server, the ebtable rule will probably fix it.

But for my case, on the squid machine, I have a web server, and
the url rewrite redirect the traffic to this web server. And I don't seem
to be able to get a reply back into squid. Which is blocking the reply 
?

[squid-users] Fw: tproxy routing issue within processes in the same machine

[Ming-Ching Tiew]

Someone claims that it is possible to do tproxy between 2 local processes.
I wonder if anyone has tested with squid. Maybe testing seems to fail ..



- Forwarded Message -
From: Balazs Scheidler 
To: Karol Piłat 
Cc: Ming-Ching Tiew ; "tpr...@lists.balabit.hu" 

Sent: Wednesday, September 19, 2012 2:47 PM
Subject: Re: [tproxy] tproxy routing issue within processes in the same machine

Hi,

This may work, the point is that the TPROXY target will not reroute
packets, so if they originally were destined to the outgoing interface,
they will continue to be so and will never cause local sockets to be
looked up.

If the packet is already routed to localhost, then it can work.

On Tue, 2012-09-18 at 20:11 +0200, Karol Piłat wrote:
> Hello,
> 
> AFAIK it is possible.
> 1. You have to bind new (spoofed) connection's port in certain range 
> (e.g. 5000 - 1, not ephemeral port range).
> 2. Setup rule to forward all outgoing TCP packets to ports in that range 
> to localhost
> 3. Make connections to physical, not loopback address.
> 
> I have it running on production for about 2 months now.
> 
> Iptables rules and routes:
>      ip rule add fwmark 1 lookup 100
>      ip route add local 0.0.0.0/0 dev lo table 100
> 
>      iptables -t mangle -N DIVERT
>      iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>      iptables -t mangle -A DIVERT -j MARK --set-mark 1
>      iptables -t mangle -A DIVERT -j ACCEPT
> 
>      iptables -t mangle -A OUTPUT -p tcp --dport 5000: -j MARK 
> --set-mark 1
> 
> Example python code to create spoofed connection:
>      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>      s.setsockopt(socket.SOL_IP, 19, 1) # IP_TRANSPARENT, not available 
> in python's stdlib
>      s.bind(('1.2.3.4', 5001))
>      s.connect(('192.168.1.9', 1234)) # connection always to physical 
> interface address!
> 
> I do free port management by myself, but you can do bind() in a loop.
> 
> Best Regards,
> Karol Pilat
> 
> W dniu 18.09.2012 10:04, Balazs Scheidler pisze:
> > Hi,
> >
> > IIRC it doesn't work for local connections/sockets, as it can't reroute
> > outgoing packets to the local interface.
> >
> > On Sun, 2012-07-29 at 06:30 -0700, Ming-Ching Tiew wrote:
> >> tproxy has  problem working within 2 processes on the same machine,
> >> ie a client process using tproxy to spoof an IP,  has problem
> >> communicating  with the server process within the same machine.
> >>
> >> It seems tproxy attaches itself to mangle table PREROUTING
> >> chain, that is unable to hook to the in-machine process. I figured
> >> that for it to work, in this case, it needs to be able to attach itself
> >> to the INPUT chain. However that hook is not supported.
> >>
> >> Is there a way to get around this problem ?
> 
> 

-- 
Bazsi

[squid-users] Create an acl src IP with matches nothing

[Ming-Ching Tiew]

How to create an acl source IP which matches nothing ?

  acl link1 src xx

What is the xx which I need to put so that it will not match
anything ?

I tried doing xx = ! 0/0. But squid complaints that it's not valid.

Reason I want to this is that I am keeping the acl parameters
in external file, I want it to matching nothing for default,
and when needed, I change through the external file for it
to match specific IPs or subnet.

Best regards.

[squid-users] Squid, tproxy, nat and multi-homed

[Ming-Ching Tiew]

I have a unique situation where I have a multi-homed
machine running squid where I will need to do some
kind of load balancing for outbound squid traffic.

Well, if both the outgoing interface are nat-ed, things will
be relatively easier, I will just do transparent proxy 
(without tproxy ). Since the identity of the original http
requests are lost anyway, tproxy will be redundant.

However, in a situation where one of the outgoing legs is 
NOT NAT-ed, while another leg is NAT-ed, this is where 
I am in trouble.

When the outgoing interface is not NAT-ed, I would like
to be able to do tproxy, retaining the identity of the
original http requests. However, when I use the squid
redirective,

http_port 3128 tproxy transparent

The un-NAT-ed leg will work just fine but I noticed that for the
NAT-ed leg, the outgoing traffic gets out to the internet
using the source IP of the original http request DESPITE that 
there is a SNAT on the nat POSTROUTING chain. As you can 
imagine, this will cause return traffic unable to come back to the 
machine.

Wonder if it is the limitation of the tproxy kernel patch,
or it's just the way I did (wrong) which causes the behaviour.

Appreciate your inputs.

 


Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

Re: [squid-users] Squid, tproxy, nat and multi-homed

[Ming-Ching Tiew]

From: "Amos Jeffries" <[EMAIL PROTECTED]>


Thanks for the quick response :-

>
> Most common failure like this requires 'you need to patch the kernel', but
> it sounds like that's been done.
>

Yupe this has been done.

> Next step is seeing what tcpdump shows about the two types of traffic.
> And possibly what type of router/balancer is doing the splitting?
>

This has been done too. Very clearly, tcpdump shows that for the
none NAT-ed leg, the identity of the original requests have been
spoofed, but the bad thing is that, it also spoofed the NAT-ed leg
as well despite there is a POSTROUTING rule to do SNAT in
the nat table. Seems to me the 'tproxy' directive in squid makes
iptables nat POSTROUTING SNAT useless !

>
> PS. Do you HAVE to use tproxy?

YES. It works if I don't use it together with nat.

> If the NATing isn't a problem you could use
> a plain intercepting/transparent proxy and have remote sources down both
> streams see the squid IP as the source of requests.
>

That will be undesirable for the none-NAT-ed leg because the traffic
will head towards an firewall will screen/filter the outgoing traffic based
on the source IPs.

Re: [squid-users] Squid, tproxy, nat and multi-homed

[Ming-Ching Tiew]

From: "Amos Jeffries" <[EMAIL PROTECTED]>
>
> No not useless. The NAT should be symmetrically unmangling any mangled
> destination on incoming traffic. As far as NAT is concerned the client is
> the real requestor. You just need to be careful that the unmangling
> happens BEFORE the tproxy return redirection toward squid.
>
> The internal side of the NAT gateway can and should be treated identical
> to the non-NAT firewall you mentioned. Both need to operate independant of
> tproxy and on the external side of any tproxy operations.
>

But the fact is that as soon as I turn on squid directive,

  http_port 3128 tproxy transparent

I will get private IP belonging to the original http web requestor
appearing
in the internet line - EVEN THOUGH - I do have a POSTROUTING
rule in the nat table to SNAT. As a matter of fact,

  iptables -t nat -nvL POSTROUTING

shows that the SNAT rule has been traversed ( and the counter is incremented
! ).

The problem goes away and everything works perfectly when I remove
'tproxy' in the squid directive !

Re: [squid-users] Squid, tproxy, nat and multi-homed

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>

>
> But the fact is that as soon as I turn on squid directive,
>
>   http_port 3128 tproxy transparent
>
> I will get private IP belonging to the original http web requestor
> appearing
> in the internet line - EVEN THOUGH - I do have a POSTROUTING
> rule in the nat table to SNAT. As a matter of fact,
>
>   iptables -t nat -nvL POSTROUTING
>
> shows that the SNAT rule has been traversed ( and the counter is
incremented
> ! ).
>

Just want to mention that my problem is fixed by doing this patch :-

http://freshmeat.net/projects/doublenatcttproxy2patch/?branch_id=71776

Regards.



Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

Re: [squid-users] load balancing traffic through squid on systems with 2 Internet connections

[Ming-Ching Tiew]

From: "Siju George" <[EMAIL PROTECTED]>
> 
> I have a System with two Internet connections.
> Is it possible to configure squid to load balance out going internet
> traffic through those two Internet Connections?
> 

This is assuming that you are running Linux :- 
Just set up multiple routing and weight assignment.
You might have to turn off kernel option which 
caches multiple routing.

Re: [squid-users] load balancing traffic through squid on systems with 2 Internet connections

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>
> 
> This is assuming that you are running Linux :- 
> Just set up multiple routing and weight assignment.
> You might have to turn off kernel option which 
> caches multiple routing.
> 

I meant MULTI PATH routing.


Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

Re: [squid-users] load balancing traffic through squid on systems with 2 Internet connections

[Ming-Ching Tiew]

From: "Siju George" <[EMAIL PROTECTED]>
> 
> Is there any option to do it in the "squid.conf" file?
> 
> I know there is a "tcp_outgoing_address" option.
> 
> just wondering if it is possible to make it use all outgoing IP
> address in a round-robin manner :-)
> 

As far as I know, you could do "split access" using 
the 'tcp_outgoing_address' method, but you can't
get squid to use it in round-robin manner.

I might be wrong. :-)

Re: [squid-users] load balancing traffic through squid on systems with 2 Internet connections

[Ming-Ching Tiew]

From: "Siju George" <[EMAIL PROTECTED]>
> >
> > But not something I would recommend. Many sites dislikes clients coming
> > from more than one IP during the same session. The client IP is often
> > embedded in session cookies etc, making the session fail if the IP
> > changes.
> >
>
> Yes Henrik.
> Such sites are identified and there is af firewall rule added to PF in
> OpenBSD to route them through the same interface.. But it is not a
> problem with majority of the sites.
>

Perhaps it will be interesting for squid to have an acl called random :-

( is there one already ? )

eg

   acl rnd random 50   # 50 %

   tcp_outgoing_address x.x.x.x rnd   <--- use x 50 % of time
   tcp_outgoing_address y.y.y.y rnd   < use y 50% of time
   tcp_outgoing_address z.z.z.z  <--- have to provide a
default in case nothing is matched

And the random acl can be used together with other acl too !

eg
acl link1 dst .
tcp_outgoing_address x.x.x.x link1 rnd

:-)

Re: [squid-users] load balancing traffic through squid on systems with 2 Internet connections

[Ming-Ching Tiew]

> If you use the
>
> http://www.openbsd.org/faq/pf/pools.html#outgoing
>
> method as I use now then even through the outgoing address will be
> changed for 50% of the packets those same packets will be routed out
> through the default interface only :-(

First of all I don't know anything about OpenBSD but we have to
know, making squid use a particular tcp_outgoing_address is as much
as what we can ask squid to do, actual routing decision should
rightfully be left to the OPERATING SYSTEM. In Linux, all these
things are addressed :-

1. You can use POLICY ROUTING - one of the capabilities of policy
routing is be able to route based on SOURCE IP instead of destination
IP.

2. You can use MULTI PATH ROUTING - ie you can start a session
based on certain relative weight you assign to a default route.

3. You can also use netfilter 'recent'/CONNMARK match - they allow
you to tie a session to an interface for a configurable amount of
time.

I might not be even exhaustive here.

All in all, my conclusion is this :- You really have to look hard to
the operating system for this instead of squid. If you have exhausted
the capabilities of OpenBSD, you would have to throw it away !

:-)

[squid-users] squid 2.6 stable13, tproxy and wccp

[Ming-Ching Tiew]

First of all the good news. I have gotten squid to work in bridge mode
and tproxy on kernel 2.6.18, squid 2.6 stable13 and uclibc 0.9.28.

And I managed to use ebtables/iptables to transparently provide
web caching.

But now the bad news, I could not get it to work using wccp, as soon as
I configure wccp_router xx.xx.xx.xx or  wccp2_router xx.xx.xx.xx, then
I will get this error in my cache.log :-

2007/07/04 14:41:34| WCCP Disabled.
2007/07/04 14:41:34| commBind: Cannot bind socket FD 14 to *:2048: (98)
Address already in use
FATAL: Cannot open WCCP Port

I have practically removed all other (networking) processes and
the problem still persist.

   # netstat -naut

( Nothing is using port 2048 )

This is how I compiled it on uclibc 0.9.28 :-

 ./configure \
--prefix=/mnt/squid \
--enable-follow-x-forwarded-for \
--enable-linux-tproxy \
--enable-linux-netfilter \
--enable-snmp

( I take it that wccp is enabled by default and I check config.log, it
indeeded said it's ON  ).

Now I search the entire web I could not find someone else having
similar problem as mine. So I am hoping someone could give some
pointers to identity the problem. I hope it's not a uclibc problem .

Appreciate it

Re: [squid-users] squid 2.6 stable13, tproxy and wccp

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>
To: 
Sent: Wednesday, July 04, 2007 3:38 PM
Subject: [squid-users] squid 2.6 stable13, tproxy and wccp


>
> But now the bad news, I could not get it to work using wccp, as soon as
> I configure wccp_router xx.xx.xx.xx or  wccp2_router xx.xx.xx.xx, then
> I will get this error in my cache.log :-
>
> 2007/07/04 14:41:34| WCCP Disabled.
> 2007/07/04 14:41:34| commBind: Cannot bind socket FD 14 to *:2048: (98)
> Address already in use
> FATAL: Cannot open WCCP Port
>

OK I have fixed the problem. I tweaked a few kernel CONFIGs and
recompile a  new kernel, it's now working. So it's not uClibc problem at
all,
but I am unsure which exactly the kernel CONFIGs which fixes the problem.
I will perhaps isolate it later as in the meantime I would like to move on
to
other things first.

Regards.

[squid-users] transparent tproxy: routing issue or my own problem ?

[Ming-Ching Tiew]

This is long I appreciate you patience.

I am using squid in a Linux box setting up as a bridge, and have
set up ebtables and iptables following the documentation
available on the Net :-

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
  --ip-destination-port 80 -j redirect --redirect-target ACCEPT

iptables -t tproxy -A PREROUTING -i br0 -p tcp --dport 80 \
  -j TPROXY --on-port 80

# this don't seem to have impact by I have put in anyway
for i in /proc/sys/net/ipv4/conf/*/rp_filter
do
 echo 0 > $i
done

On a brief glance it seems it's working properly but upon detail
investigation,
there are some issues.

This is my observation :-

If I place the Bridge/Squid S in a subnet A  before the default internet
gateway D, then all the machines inside the same subnet A can be
serviced by the squid cache engine. Sniffing confirmed that the source
IP has been spoofed by Bridge/Squid S.

However, if there is a subnet B, which is connected to subnet A, via
a router R, then all the machines inside subnet B will have problem
getting the http reply packets but http request packets have no
problem going out.

Note that none-http packets because it has not been redirected by the
ebtable rules, have no problem at all. This shows that the routing
outside of the Bridge/Squid, have all been set up correctly.

Then I added a route inside the Bridge/Squid S for the subnet B via
router R, then the web request/reply problem is solved.

It seems then to me that the http reply ( source port 80 ) has also be
directed ***INTO*** the Bridge/Squid S. Why is that so ? Why didn't the
Bridge/Squid forward the reply packet to the other side of the
interface ?

I am looking for something more transparent. Any insight is much
appreciated.

p/s :-

The logs I capture using tcpdump on the squid machine before and after I
added the route. Network B 10.6.1.0/24, Network A 192.168.128.0/18,
Router R  10.6.1.1<-->192.168.128.50,  Squid 192.168.128.20.

Before :-

squid:~> tcpdump -ni br0 host 10.6.1.2 and port 80
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 68 bytes
09:06:12.974206 IP 10.6.1.2.39895 > 192.168.128.20.80: S
3302818155:3302818155(0) win 5840 
09:06:12.974252 IP 66.249.89.99.80 > 10.6.1.2.39895: S
3648928734:3648928734(0) ack 3302818156 win 5792 
09:06:15.974464 IP 10.6.1.2.39895 > 192.168.128.20.80: S
3302818155:3302818155(0) win 5840 
09:06:15.974492 IP 66.249.89.99.80 > 10.6.1.2.39895: S
3648928734:3648928734(0) ack 3302818156 win 5792 
09:06:16.233344 IP 66.249.89.99.80 > 10.6.1.2.39893: S
3551948981:3551948981(0) ack 3215288824 win 5792 
0


squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:03:46.982444 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 
09:03:49.982585 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 
09:03:50.334072 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0)

squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:03:46.982444 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 
09:03:49.982585 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 
09:03:50.334072 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0)


After I added a route :-

squid:~> ip route add 10.6.1.0/24 via 192.168.128.50

squid:~> tcpdump -ni br0 host 10.6.1.2 and port 80
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 68 bytes
09:12:55.957274 IP 10.6.1.2.47574 > 192.168.128.20.80: S
3726051898:3726051898(0) win 5840 
09:12:55.957398 IP 66.249.89.147.80 > 10.6.1.2.47574: S
4058179260:4058179260(0) ack 3726051899 win 5792 
09:12:55.95 IP 10.6.1.2.47574 > 192.168.128.20.80: . ack 4058179261 win
92 


squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:12:55.962016 IP 10.6.1.2.43328 > 66.249.89.99.80: S
4071804540:4071804540(0) win 5840 
09:12:56.403123 IP 66.249.89.99.80 > 10.6.1.2.43328: S
3907206245:3907206245(0) ack 4071804541 win 8472 

squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80 tcpdump: WARNING: eth0:
no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full 

Re: [squid-users] transparent tproxy: routing issue or my own problem ?

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>
>
> I am using squid in a Linux box setting up as a bridge, and have
> set up ebtables and iptables following the documentation
> available on the Net :-
>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
>   --ip-destination-port 80 -j redirect --redirect-target ACCEPT
>
> iptables -t tproxy -A PREROUTING -i br0 -p tcp --dport 80 \
>   -j TPROXY --on-port 80
>
>
> On a brief glance it seems it's working properly but upon detail
> investigation,
> there are some issues.
> 
> I am looking for something more transparent. Any insight is much
> appreciated.


I think I fixed the issue by changing the ebtables rule to :-

ebtables -t broute -A BROUTING --logical-in br0 -p IPv4 --ip-protocol 6 \
   --ip-destination-port 80 -j redirect --redirect-target DROP

Note that subtle changes. With that I don't need to add routes and other
shits.
I would appreciate feedback from others to see if this is a better rule than
the original one.

Regards.

Re: [squid-users] transparent tproxy: routing issue or my own problem ?

[Ming-Ching Tiew]

> I think I fixed the issue by changing the ebtables rule to :-
>
> ebtables -t broute -A BROUTING --logical-in br0 -p IPv4 --ip-protocol 6 \
>--ip-destination-port 80 -j redirect --redirect-target DROP
>
> Note that subtle changes. With that I don't need to add routes and other
> shits.
> I would appreciate feedback from others to see if this is a better rule
than
> the original one.
>

Sorry false alarm. The new rule bypasses all traffic from squid, that's why
it is working. Back to square ones. Need to work harder on it.

:-(

[squid-users] Re: transparent tproxy: routing issue or my own problem ?

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>
>
> It seems then to me that the http reply ( source port 80 ) has also be
> directed ***INTO*** the Bridge/Squid S. Why is that so ? Why didn't the
> Bridge/Squid forward the reply packet to the other side of the
> interface ?
>
> I am looking for something more transparent. Any insight is much
> appreciated.
>

Sorry for taking up your bandwidth it looks like I am looking for something
impossible at this moment.

The http reply has to go back **INTO** the Bridge/Squid box, so that it can
make
a cache copy, as such the http reply to the http request will have to ROUTE
out
from the bridge/squid box ( verses  BRIDGE ).

Unless some enhancement is made to do some kind of "connection tracking",
and thus reply the packet back to the mac address of the original requests.

Regards.

Re: [squid-users] transparent tproxy: routing issue or my ownproblem ?

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
> -i eth0 --ip-source your.lan.network/mask \
> --ip-destination-port 80 -j redirect --redirect-target ACCEPT

If you look at the http://ebtables.sourceforge.net/examples.html#easy,
it says when re-direct on ethX, it should be DROP instead of accept,
while doing it on brX, then it should be ACCEPT. I am no ebtables
expert, correctly if I am wrong. :-)

> If you are to use TPROXY then I'd recommend using the bridge-netfilter
> integration instead of ebtables.

I lost you, what do you mean by bridge-netfilter integration. Any URL ?

> This because TPROXY needs to intercept
> the return traffic as well, not just lan->internet traffic. It's
> possible to add ebtables rules for this by doing rules inverse to the
> above.
>
>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
> --ip-destination your.lan.network/mask \
> --ip-source-port 80 -j redirect --redirect-target ACCEPT
>

Hmmm interesting. I do not  have this rule in my system and I am
able to surf the NET via the bridge/squid ( if I set up proper routing ).
Now you make me wonder if I have set it up correctly. It seems to 
me that the internet-->lan traffic is already heading into the bridge, 
so there is no need to hijack it again. Am I missing something ?

Regards.

Re: [squid-users] transparent tproxy: routing issue or myownproblem ?

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

>
>> I lost you, what do you mean by bridge-netfilter integration. Any URL ?
>
> It's a kernel option.

Did you mean

CONFIG_BRIDGE_NETFILTER=y

and all these :-

#
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m

I have plenty of those inside many kernel and modules. How do I use it
instead of TPROXY ?

>> Hmmm interesting. I do not  have this rule in my system and I am
>> able to surf the NET via the bridge/squid ( if I set up proper routing ).
>
> It will work fine until you use TPROXY to have Squid fake the source IP
> on the requests it sends..

As far as I can tell my system is already faking the source IP. But I might
be
wrong. :-)

Do you mean it is a result of some of the kernel CONFIGs which I had instead
of TPROXY module ?

Regards.

[squid-users] cachemgr.cgi - display raw text

[Ming-Ching Tiew]

I am using squid2.6 stable13, kernel 2.6.18 and tproxy and
uclibc 0.9.28.

I am using httpd from busybox 1.4.2, running at port 8080
since squid's http port is 80.

The problem I have is that when I run cachemgr.cgi,

 http://192.168.128.20:8080/cgi-bin/cachemgr.cgi

it (correctly) display the formatted html text asking
me for name and password, and but when I click
continue it displayed this text on the browser rather than
formatted html, it looks to me there are extra text in front
of the output which confuses the browser.

Status: 200 200 OK
Server: squid/2.6.STABLE13
Date: Thu, 12 Jul 2007 01:36:02 GMT
Expires: Thu, 12 Jul 2007 01:36:02 GMT
Last-Modified: Thu, 12 Jul 2007 01:36:02 GMT
X-Cache: MISS from squid.redtone.com
Via: 1.0 squid.redtone.com:80 (squid/2.6.STABLE13)
Proxy-Connection: close
Content-Type: text/html

http://www.w3.org/TR/html4/loose.dtd";>
[EMAIL PROTECTED]: menu


Cache Manager menu for localhost:
Memory Utilization
..

Re: [squid-users] cachemgr.cgi - display raw text

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <[EMAIL PROTECTED]>
> 
> it (correctly) display the formatted html text asking
> me for name and password, and but when I click
> continue it displayed this text on the browser rather than
> formatted html, it looks to me there are extra text in front
> of the output which confuses the browser.
> 

I worked around the problem by doing this in a new cgi :-

# cat cmgr.cgi

#!/bin/sh
/mnt/squid/libexec/cachemgr.cgi | sed -e '1,8d'

Everything works perfectly after this.

Regards.

Re: [squid-users] cachemgr.cgi - display raw text

[Ming-Ching Tiew]

>> #!/bin/sh
>> /mnt/squid/libexec/cachemgr.cgi | sed -e '1,8d'
>> 
>> Everything works perfectly after this.
>
>Odd.. what kind of web server are you using?

I use busybox httpd. Do you mean the extra text is expected
and it should be handled by the web server ? Then it might
then be the web server configuration problem.

But I have a bigger problem now. I noticed that after running
the cachemgr.cgi, I will get this in my cache.log :-

WARNING! Your cache is running out of filedescriptors

If I don't run it, it's fine.

I have 1 G RAM, running nothing else important except squid.
And I allocated 128 M for cache_mem. I also allocated
a huge cache_dir :-

cache_dir ufs /mnt/squid/var/cache 4 16 256

Is it that the cache_dir is too big ?

Regards.

[squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

Anyone has experience with level 4 switch  ? What is the working 
principle of a level 4 in respect to redirecting web traffic to a cache
engine ? Does it do dst IP address rewrite ( iptables DNAT ) or 
does it do dst MAC address rewrite ( ebtables dnat ) when redirecting 
traffic to the cache engine ?

Can I simulate a level 4 switch behaviour using Linux ? If yes,
any insight to the necessary ebtables/iptables rules ?

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Adrian Chadd" <[EMAIL PROTECTED]>

> On Tue, Jul 17, 2007, Ming-Ching Tiew wrote:
> >
> > Anyone has experience with level 4 switch  ? What is the working
> > principle of a level 4 in respect to redirecting web traffic to a cache
> > engine ? Does it do dst IP address rewrite ( iptables DNAT ) or
> > does it do dst MAC address rewrite ( ebtables dnat ) when redirecting
> > traffic to the cache engine ?
>
> You need to be more specific when you say "Layer 4 switch", as how it
> does things depends entirely on the switch.
>
> Cisco TCAM switches with L3 functionality will want to do L2 rewrite.
> It'll rewrite the source/destination MAC address and send the packet off
> to the cache for (potential) interception.
>

I got the word "level 4 switch" from someone who knows nothing else,
so I have to make guesses too. But I would imagine that it is doing L2
rewrite.
Care to elaborate how the L2 rewrite in Cisco TCAM works ?

Does it constantly do arp request based on IP of the cache engine and update
the MAC address accordingly ( much like ip routing does ) or it statically
configure the the mac address of the cache engine ?

Regards.

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

>> Can I simulate a level 4 switch behaviour using Linux ? If yes,
>> any insight to the necessary ebtables/iptables rules ?
>
>Linux policy routing is an example of "layer 4".
>
>For load balancing see Linux Virtual Server / IPVS. Part of the linux
>kernel, and performs most of the forwarding choices you find in "layer 4
>switches". http://www.linuxvirtualserver.org/

Thank you and this is the kind of answer I am looking for.

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

>> Can I simulate a level 4 switch behaviour using Linux ? If yes,
>> any insight to the necessary ebtables/iptables rules ?
>
>Linux policy routing is an example of "layer 4".

I am wondering if this setup shall be a reason representation of a so-called
level 4 bridge. This configuration works under both 'tproxy transparent'
as well as 'transparent' mode for squid 2.6 stable 13.

Assuming :-

NETMASK=255.255.192.0
SQUID_IP=192.168.128.50
L4_SWITCH_IP=192.168.128.51
INTERNET_GW=192.168.128.1

1. On the L4 switch create bridge br0 consisting of 3 ethernet interfaces.

eth1 is connected to internet
eth0 is connected to inside network
eth2 is connected to squid

# ifconfig eth0 0.0.0.0 promisc up
# ifconfig eth1 0.0.0.0 promisc up
# ifconfig eth2 0.0.0.0 promisc up
# brctl addbr br0
# brctl addif br0 eth0
# brctl addif br0 eth1
# brctl addif br0 eth2
# ifconfig br0 $L4_SWITCH_IP netmask $NETMASK up

2. Set up the bridge to mark the packets so that policy routing works :-

   from inside network go to internet destination port 80, mark 1.
   from internet come back with source port 80, mark 1 as well.

   # ebtables -t broute -A BROUTING -i eth0 -p IPv4 --ip-protocol 6 \
  --ip-destination-port 80 -j redirect --redirect-target DROP
   # iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 \
   -j MARK --set-mark 1

   #ebtables -t broute -A BROUTING -i eth1 -p IPv4 --ip-protocol 6 \
--ip-source-port 80 -j redirect --redirect-target DROP
   # iptables -t mangle -A PREROUTING -i eth1 -p tcp --sport 80 \
   -j MARK --set-mark 1

3. Set up additional routing table and ip rule :-

# echo '100 one' > /etc/iproute2/rt_tables
# ip rule add fwmark 1 lookup one
# ip route add default via $SQUID_IP table one

( routing table 'one' need only to have one line, ie the default route,
local interface routes will interfere with tproxy  )

# ip route add default via $INTERNET_GW table main

Regards.

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Michel Santos" <[EMAIL PROTECTED]>
>
> aren't you mixing things here? *layer* 4 and *level* 4 are different
> things and policy routing eventually is still another
>

I know you are the expert but your answers are not helping at all.

I don't need to be told that you are the expert but I will be glad
to be told how different and in what way they are different.

>
> for policy routing you do not need a level 4 bridge neither a level 4
> switch because any OS with any kind of forwarding capable firewall package
> can do that and in order to do routing (any) you do not need a bridge
> setup at all
>

I was just trying to slip in a box which does things transparently.
I intend to get a little further than this, I want to even add gre to it
so then it will hopefully behave like a Cisco doing WCCP2 with an
external squid box with wccp2 configured.

Purpose is modest :- Use it to check if the squid  is set up correctly
without disturbing existing network.

Maybe you could be a little more specific about if you were to do it,
how would you go about doing it. More specifically when the
squid is 'tproxy transparent', ie when the forward path is spoofed,
how to you handle the routing of the return path.

Regards.

Re: [squid-users] Squid and level 4 switch

[Ming-Ching Tiew]

From: "Michel Santos" <[EMAIL PROTECTED]>
> 
> anyway, level 3 switch/bridge understand up to OSI Layer4 and layer 4
> switch/bridge understand up to OSI layer 7 as I said already before
> 
> so you can google for "OSI Layer definition" and see what that is, that
> are the differente network layers from hardware up to application layer
> 

I don't need more text book answers. Anyway many thanks for your attempt. 
This thread is considered closed.

Regards.

[squid-users] forward and reverse proxy - the difference

[Ming-Ching Tiew]

Believe it or not, I got problem understanding the basics.

What's the difference between forward and reverse proxy.

When I read the article, 

http://jayant7k.blogspot.com/2006/10/reverse-proxy-using-squid.html

When I read paragraph 3,4 & 5, I think what is said about 
reverse proxy is equally applicable to forward proxy. Is there
a simpler way to explain the difference between the two ?

But of course, for forward proxy, I would not need to configure cache_peer.
So why is there a need for 'vhost' and 'vport' directive ?

Regards.

[squid-users] Increasing file descriptor and others

[Ming-Ching Tiew]

First of all the good news. After much struggle, I finally managed to
get Squid 2.6 stable 13 to work with Foundary ServerIron XL with this
config :-

http_port 3128 tproxy transparent
http_port 80 vhost

It seems the second line is a must for whatever reasons.

And iptables  :-

 iptables -t tproxy -A PREROUTING -i $DEVICE -p tcp --dport 80 \
 -j TPROXY --on-port 3128

But now with my squid compiled with 4096 file descriptor and aufs is
running out of file descriptor ( CPU loading appears to be still healthy ).
Now I would want to increase things to :-

  --enable-async-io=24
  --with-maxfd=1

Do they look like reasonable figure ? Any danger for increasing these values
?

Regards.

[squid-users] Squid and PPPoE - peculiar things

[Ming-Ching Tiew]

Anyone has experience peculiar things with Squid and PPPoE ?

I have a setup where Squid is doing transparent tproxy for PPPoE
and non-PPPoE users, however the experience is that when
squid is serving the cached files for PPPoE users, it's slower
than a commercial product.

Is it possible that this is a MTU problem ? Does it make sense
to change the ethernet interfaces to have a smaller MTU 
( matching with a typical PPPoE config ) ? Or do I have to
add iptables rule to clamp-mss-to-pmtu ? 

I have already got 

  httpd_accel_no_pmtu_disc off

as the default value.

Regards.



Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

[squid-users] What is the price to pay for file descriptor ?

[Ming-Ching Tiew]

What is the price to pay for increasing the file descriptor ?

Has anyone compiled squid with 50,000 file descriptor ?

I am using it on a machine with 2 G RAM and SCSI Harddisk.

Regards.


Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

[squid-users] I/O tests and squid

[Ming-Ching Tiew]

I wonder if anyone has a good I/O test which will sort of represent
the way squid needed the I/O to perform. Basically I need one program
which I can use to check the influence of various components of the
system ( OS, parameters, harddisk, library version ) on the I/O for
maximizing
squid performance.

For example, when I run on the ***SAME*** machine, one with a Fedora
Core 5 ( kernel 2.6.21.3 ) and one with uclibc-0.9.28 ( kernel 2.6.18 ),
using bonnie-1.4 ( a very old program and maybe buggy with respect to
the big size files ),  this is the result :-

FC5 :
# ./Bonnie -s 2004 -d tmp
Bonnie 1.4: File 'tmp/Bonnie.24172', size: 2101346304, volumes: 1
Writing with putc()... done:  35399 kB/s  93.7 %CPU
Rewriting...   done:  12557 kB/s   5.0 %CPU
Writing intelligently...   done:  37587 kB/s  14.4 %CPU
Reading with getc()... done:  14878 kB/s  86.6 %CPU
Reading intelligently...   done:  27846 kB/s  -0.0 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
  ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd
Seek-
  -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k
(03)-
MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec
%CPU
testin 1*2004 35399 93.7 37587 14.4 12557  5.0 14878 86.6 27846 -0.0
 130.3 -0.0

uclibc 0.9.28
# ./Bonnie -s 2004 -d tmp
Bonnie 1.4: File 'tmp/Bonnie.19932', size: 2101346304, volumes: 1
Writing with putc()... done:  44332 kB/s  35.9 %CPU
Rewriting...   done:  18676 kB/s  11.3 %CPU
Writing intelligently...   done:  49168 kB/s  18.5 %CPU
Reading with getc()... done:  39758 kB/s  30.5 %CPU
Reading intelligently...   done:  44955 kB/s  13.8 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
  ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd
Seek-
  -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k
(03)-
MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec
%CPU
cnxsq  1*2004 44332 35.9 49168 18.5 18676 11.3 39758 30.5 44955 13.8  116.6
0.7

It seems that the I/O on uclibc-based system runs better than a FC5 system.
This is will opposite to my expectation. I am not sure if the conclusion
will
be applicable to running squid.

However, when I run the bonnie++-1.93c ( statically compiled version since
the
said uclibc system does not have C++ runtime ), the performance of both
configurations are almost identical.

Appreciate comments.



Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

Re: [squid-users] I/O tests and squid

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

Re: [squid-users] I/O tests and squid

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>
>
>
> Shouldn't have much effect on Squid as Squid is using direct POSIX I/O,
>and not C "stdio" I/O, bypassing almost all of the C library.

That's why I asked the question, what will be the test which I can
run to verify the performance of IO needed by squid. Because
when I tweak something, I need to quickly verify it instead of
putting it to life use.

I am having a system which handles about 9000 active sessions
now, and the iostat result is as such :-

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0.92   0.001.09  6.16   0.0091.83

Device:tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda  67.86   393.10  1233.41   27988252   87818678
sdb  48.64   152.93   999.94   10888386   71195106

As shown above everything the system is hardly stress in terms of
CPU. Everything is blocking at I/O - which I am not sure if it is
disk IO or network IO. And it seems the io wait figures is building
up. I am worried that it will continue to build up and causing bottle
necking.




Important Warning! 

*** 

This electronic communication (including any attached files) may contain 
confidential and/or legally privileged information and is only intended for the 
use of the person to whom it is addressed. If you are not the intended 
recipient, you do not have permission to read, use, disseminate, distribute, 
copy or retain any part of this communication or its attachments in any form. 
If this e-mail was sent to you by mistake, please take the time to notify the 
sender so that they can identify the problem and avoid any more mistakes in 
sending e-mail to you. The unauthorised use of information contained in this 
communication or its attachments may result in legal action against any person 
who uses it.

Re: [squid-users] I/O tests and squid

[Ming-Ching Tiew]

From: "Henrik Nordstrom" <[EMAIL PROTECTED]>

>>  avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>0.92   0.001.09   6.16   0.00
91.83
>
>
> It's not much blocking on disk I/O either, only 6.16%. 91.83% of the
> time your server is doing absolutely nothing.

The said computer has two 4-core CPU, which is registered as 8 processors
to the Linux. Perhaps that's why the CPU utilization is registered as low.

> > And it seems the io wait figures is building
>> up. I am worried that it will continue to build up and causing bottle
>>  necking.
>
>There is a significant increase in disk I/O transactions when the cache
>has been filled and Squid starts to recycle space. Then it levels out
>and stays relative to the amount of cachable traffic you have.

You are obsolutely right about that observation, unfortunately I did not
know how to deal with this transition and users are complaining slow
http response and I had to put Squid off-line. From being able to
handle 9000 requests concurrently, I don't know for want changes I made
( or for what reasons ), it reduced to 1300 requests and I had to
finally retire Squid.  Whereas somehow the other commercial unit is
able to somehow hand 15000 server request concurrently. Sad that I
could not make it to using squid.

Another problem I see is that a typical service provider cache engine
will get considerable amount of DoS/syn-flood attacks ( at port 80 ).
Netfilter connection tracking becomes double edge sword. Perhaps I
did not plan out a good scheme to deal with that from the start.

Best regards.

Re: [squid-users] I/O tests and squid

[Ming-Ching Tiew]

From: "Adrian Chadd" <[EMAIL PROTECTED]>
> 
> Did you try COSS?
> 
> Commercial units have had a lot more attention. Chances are you've
> not gotten someone with Squid expertise to se any of your stuff up
> or do any deep analysis of the problems; what did you expect
> would happen?
> 
> 

Is there anyone who can provide the service ( with a fees of course ) ?
Send me private mails if there is anyone happened to be in this list.

Regards.