Some assumptions:
1) You are using a Cisco Router to redirect traffic to the squid box via
WCCP
2) 12.4(15)T8 or higher IOS on the router
3) In my setups, the squid box is always Layer 2 adjacent to the Cisco
router, either through a dedicated interface, or a sub-interface.
4) The ability to compile and install a Linux kernel. Please note that
in these steps, I am NOT using a redhat kernel, nor am I using the
RedHat method of building a kernel.
5) Some steps outlined here can be achieved through several different
means, follow the steps exactly before emailing me or the list, as I
have tested other methods, and they don't always work (case in point:
GRE tunnel interface creation.)
6) This setup assumes a separate WCCP service group for each direction
of the HTTP connection, this is not needed, but makes the setup more
scalable. If you choose to do it a different way, then YMMV.
In the kernel build specific steps, I actually include possibly to much
information, as well as tell you to enable things that are not always
needed for TPROXY related functionality, or never related to TPROXY
functionality. I included them because they fit more environments, and
thus less time wasted by people asking me questions, not that I mind but
I don't have enough time to answer all the emails I get. I tried to
prepare this information out without errors, if the steps don't work,
email me with the details of where you had problems so that I can adjust
the steps below. At the end the steps below are some common things to
watch for in the steps that can cause the setup not to work.
Steps
1) Install CentOS 5.3, make sure you install nothing other than the base
packages, and trim even those down. I tend to install specific packages
from the distro later.
Note: I suggest that you make separate partition(s) for where squid will
actually store its caches. Later mount these partitions with specific
options (like "noatime") that will help increase performance.
2) In the initial ncurses-based setup screen, turn off services that you
don't need, and turn off selinux compeletely.
3) After install and initial bootup and configuration, run "yum update"
to update the system for fixes, etc. Then reboot.
4) After step 2, issue this yum command:
yum install libcap libcap-devel gcc gcc-c++ bison flex yacc autoconf
automake ncurses ncurses-devel rpm-devel libpcap tcpdump
Note: let it install other dependency packages. The command above
installs compiles, utilities, etc.
5) Download iptables-1.4.3.2 from netfilter.org
6) Download kernel 2.6.30 from kernel.org
7) Download squid-3.1.0.8 from squid.org
8) Decompress the kernel source, I decompress it to /usr/src/, although
I have read all over the place that this is a bad thing to do. The
location really does not have to be /usr/src/
9) Go into the kernel source directory, issue the following command:
cp /boot/config-2.6.18-128.1.14.el5 ./RH-config-boxed.config
10) Issue this command: make menuconfig
11) When the ncurses-based kernel config screen loads, select the "Load
an Alternate Configuration File" and type in the full path to the
RH-config-boxed.config. This will load the current kernel config, and
there may be some errors, all of which can be ignored.
12) Configure the kernel as you normally would, but be sure to enable
the following:
In "Networking support -> Networking options"
Enable (not as modules):
Packet socket
Packet socket: mmapped IO
TCP/IP networking
IP: advanced router
IP: policy routing
Enable (as modules):
IP: tunneling
Enable (not as modules):
IP: GRE tunnels over IP
IP: broadcast GRE over IP
Network packet filtering framework (Netfilter)
In "Networking support -> Networking options -> Network packet filtering
framework (Netfilter)"
Enable (not as modules):
Advanced netfilter configuration
In "Networking support -> Networking options -> Network packet filtering
framework (Netfilter) -> Core Netfilter
Configuration"
Enable (as modules):
Netfilter connection tracking support
Enable (not as modules):
Connection tracking security mark support
Connection tracking events
Enable (as modules):
Connection tracking netlink interface
Transparent proxying support (EXPERIMENTAL)
Netfilter Xtables support (required for ip_tables)
"CONNMARK" target support
"MARK" target support
"TPROXY" target support (EXPERIMENTAL)
"connmark" connection mark match support
"conntrack" connection tracking match support
"mark" match support
"socket" match support (EXPERIMENTAL)
"state" match support
In "Networking support -> Networking options -> Network packet filtering
framework (Netfilter) -> IP: Netfilter Configuration"
Enable (as modules):
IPv4 connection tracking support (required for NAT)
IP tables support (required for filtering/masq/NAT)
Full NAT
MASQUERADE target support
REDIRECT target support
Packet mangling
13) After setting the above options, and any other items you want, exit
out of the kernel config, saving your changes. It will save the kernel
compile config to RH-config-boxed.config so issue the following command
to put the new config in the right
place:
cp RH-config-boxed.config config-centos
Then do the make, make_install_modules, make install, if no errors,
adjust grub.conf to boot to the new kernel. The reboot to
the new kernel.
14) Assuming the kernel compiled, installed and booted properly, it is
time to update iptables. Decompress the iptables
source, and use the following configure command:
./configure --enable-devel --enable-libipq --bindir=/bin --sbindir=/sbin
--sysconfdir=/etc --with-kernel=<path to new kernel source dir>
--with-kbuild=<path to new kernel source dir> --with-ksource=<path to
new kernel source dir>
Then do the make, make install
15) Edit ld.so.conf to add a library path:
vim /etc/ld.so.conf
add a line to the end of the file: /usr/local/lib
16) I suggest rebooting, just to make sure that iptables upgrade is
working. Do a reboot, then do a "service iptables status" to make sure
that iptables is running fine. If it is not running ok, it will show
either a status failed, or an empty rule set.
17) Assuming that the iptables upgrade is working fine, the next steps
are to add the rules and interfaces needed for WCCP and TPROXY
functionality. First, issue a "insmod ip_gre", then do a "dmesg | tail"
and make sure that you see "GRE over IPv4 tunneling driver" if you don't
see anything, or you get an error from insmod, you either compiled GRE
not as a kernel module, or not at all.
18) Create the GRE tunnel interface, issue "ifconfig gre0 <ip address of
squid server> netmask <netmask of squid server ip> up"
19) Issue an "ifconfig" and make sure you see a gre0 interface.
20) Next we need to add iptables rules to allow traffic to the gre
interface itself, gre protocol traffic across the Ethernet interface,
and WCCP traffic from the router. We will do this by editing the
iptables save file directly. Do the following:
a.) service iptables save
b.) vim /etc/sysconfig/iptables
c.) just after the line that says ":RH-Firewall-1-INPUT - [0:0]" add the
following:
-A INPUT -i gre0 -j ACCEPT
-A INPUT -p gre -j ACCEPT
-A INPUT -i eth0 -p gre -j ACCEPT
d.) Somewhere lower in the file where the "-A RH-Firewall-1-INPUT" rules
are, add the following:
-A RH-Firewall-1-INPUT -s <address of cisco router>/32 -p udp -m
udp --dport 2048 -j ACCEPT
NOTE: The "<address of cisco router>" in this instance is the ip
address of the router that the squid box itself uses.
Port 2048 is the port that WCCP traffic uses.
e.) save the file and exit vim. Do a "service iptables restart" to make
sure you don't get any errors.
21) Next we add the TPROXY related iptables rules, from the command line
prompt issue the following commands:
iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 --on-port <port squid listens to> --on-ip
<squid server ip>
22) Assuming the commands in step 21 didn't give errors, do a "service
iptables save"
23) We need to edit the iptables rule order, to make sure it is in the
correct order: "vim /etc/sysconfig/iptables" and make sure the section
at the bottom of the file, from the ":DIVERT - [0:0]" onward looks
something like:
-A DIVERT -j MARK --set-xmark 0x1/0xffffffff
-A DIVERT -j ACCEPT
-A PREROUTING -p tcp -m socket -j DIVERT
-A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port <squid port>
--on-ip <squid server ip> --tproxy-mark 0x1/0x1
When doing the cli commands in step 21, iptables puts the rules in the
wrong order for them to work for TPROXY.
Save the file and exit vim.
24) do a "service iptables restart"
26) Issue the following commands, per the TPROXY wiki article:
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
If no errors from the two commands, add them to the end of
/etc/rc.d/rc.local
27) I'm not sure this is needed anymore, but it doesn't seem to break
anything, so add the following line to the end of /etc/rc.d/rc.local:
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
28) In the beginning of the rc.local file, add the following two lines:
modprobe ip_gre
<the ifconfig gre0 command line from step 18 above>
29) Edit /etc/sysctl.conf, make sure there is a line somewhere that
says:
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
30) Reboot the server
31) Next, build you squid 3.1.x.x, make sure that pass
"--enable-linux-netfilter" to the configure directive, build and install
squid.
32) You will need to edit the squid config to make sure that it has what
it needs to use TPROXY and WCCP. For tproxy, you need one item, the
"http_port" directive needs tproxy appended to it, I do the following:
http_port <ip of squid server>:<port to bind to, "squid port" from step
23 above> tproxy disable-pmtu-discovery=always
33) Configure the WCCP specific related items in the squid config file,
specifically:
a) wccp2_router <router address used in the steps above>
b) wccp_version 2
c) wccp2_rebuild_wait on
d) wccp2_fowarding_method 1
e) wccp2_return_method 1
f) wccp2_assigment_method 1
g) wccp2_service dynamic 80
h) wccp2_service dynamic 90
i) wccp2_service_info 80 protocol=tcp flags=src_ip_hash
priority=240 ports=80
j) wccp2_service_info 90 protocol=tcp
flags=dst_ip_hash,ports_source priority=240 ports=80
34) There are other squid configuration items that are need which are
not included here because they are not specific to WCCP/TPROXY
functionality and the specifics vary by environment.
35) In the router configuration, you need to do at least the following:
a.) enable wccp globally with (this might actually be optional):
ip wccp web-cache
b.) enable the specific services:
1.) ip wccp 80
2.) ip wccp 90
Note: I use a redirect list ACL with the two commands
above so that the router doesn't WCCP redirect specific web sites. The
command would look like "ip wccp 80 redirect-list 122", and access-list
122 would be a list of denies for sites to not be redirected, and a
permit any any at the end of the access-list to allow all other websites
to be wccp redirected.
c.) On the interface that is adjacent to the squid box, do a "ip
wccp redirect exclude in" this command makes it so that the router does
not redirect the squid traffic as well as other client traffic.
Some things to check along the way:
1) Make sure that the GRE interface on the squid box is seeing packets
coming in. You should never see any packets going out the GRE interface,
and you will only see packets coming in the GRE interface after the WCCP
process on the router redirects them to the squid box.
2) Keep in mind that you should separate the idea in your mind between
WCCP traffic and tunneled HTTP traffic. The router and the squid process
talk to each other for status and service information with WCCP. HTTP
traffic received from the client by the router for redirect is
encapsulated in GRE (to preserve it) and then forwarded to the GRE
interface.
3) The global command "show ip wccp" is a useful router command. In the
output of this command, you should see two "Service Identifier" sections
(one for 80, one for 90 if you use my setup steps.) Within each "Service
Identifier" group, the number of service group clients and routers
should each be 1. If they are not, them some facet of WCCP conversation
between the router and the squid server is not working. Check the
iptables port 2048 setup step 20 part d above.
4) wccp event debug commands are useful. Use "debug ip wccp events" and
set you router logging to debug level to see "Hello" and "Here_I_am"
packets. You need to make sure you see both to insure that the router
and squid box are WCCP talking.
5) Surf the Web from a client and see if you get to a web site. If it
doesn't work, check the items above, recheck the steps.
It if does work, go to a site that tells you your IP to make sure TPROXY
is working and using the client IP and not the squid IP. It is entirely
possible that the whole setup will work, but not the client IP spoofing.
I also suggest that you burn the setup in with web surfing to make sure
it doesn't break.
I am interested to here people's feedback so that I can improve the
steps above, as well as share optimizations.
Nick
