Re: [squid-users] Not able to access Thunderbird from a linux client through squid

2009-09-28 Thread Avinash Rao 
On Fri, Sep 18, 2009 at 12:17 PM, Amos Jeffries  wrote:
> Avinash Rao wrote:
>>
>> Hi,
>>
>> I am using squid2.6stable18 on ubuntu 8.04 server.
>> I have configured squid for very basic proxy and my squid.conf is below.
>> I am not able to access thunderbird email through this proxy
>> configuration, I am using thunderbird from a Ubuntu client, but i am
>> able to access internet using Mozilla Firefox browser, but not
>> thunderbird. How can i get this working?
>>
>> The thunderbird client uses Port 110 and 25 to access emails and i
>> have enabled them here.
>
> Thunderbird does not use HTTP proxy for email fetching.
>
> It will only use HTTP proxy settings for fetching HTML content images,
> videos and virus code which are embeded.
>
> The emails themselves are fetched via native POP3 or IMAP protocols.
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE6 or 3.0.STABLE19
>  Current Beta Squid 3.1.0.13
>

I understand, but why isn't it working? If the machine has direct
connection to internet (modem connected to the machine) thunderbird
works, but if it has to go through proxy it doesn't work.

Re: [squid-users] squid.conf and Squid 2.6 vs. Squid 2.7

2009-09-28 Thread Amos Jeffries 
On Tue, 29 Sep 2009 00:10:35 -0400, Michael Lenaghan 
wrote:
> I used a Python recipe from PyPi to build, install and configure
> Squid. The same recipe, config, etc. breaks when moving from Squid 2.6
> to Squid 2.7. Other people have reported seeing the same error but no
> one has bothered to track down a fix. I have one now, but I really
> only got it by trying a bunch of different things. Before reporting
> the fix back to various groups I'd like to understand what broke and
> why the change made a difference.
> 
> In my particular case Squid was being used as a reverse (caching)
> proxy in an Nginx -> Squid -> Zope chain. Squid was running on port
> 10932 and Zope on 45086. With 2.6 everything worked, but when
> switching to 2.7 Squid's port number became visible in URLs. For
> example, in the main page the href to the logo appeared as:
> 
> http://[domain]:10932/logo.jpg
> 
> rather than:
> 
> http://[domain]/logo.jpg
> 
> I've pasted the original conf file below (with domain names and such
> abstracted out), but the fix involved the addition of vport=80 to the
> http_port line:
> 
> http_port 127.0.0.1:10932 accel vhost vport=80 defaultsite=[domain]
> 
> I've had a very difficult time finding good docs for vhost, vport and
> defaultsite. I've looked and I've searched in many places, but I
> haven't found anything that would help me explain *why* this change
> worked.

Looks like the cleanup made vport= replace the Host: header port. vport by
itself lookup and use the one already there.

>From the commit message the fix makes defaultsite= optional when
visible_hostname is set to the wanted value. This simplifies your config a
little.

Ref: 
http://www.squid-cache.org/Versions/v2/2.7/changesets/12051.patch
http://bugs.squid-cache.org/show_bug.cgi?id=2192
(comment 4 and 5 describes the options and how they work now).

> Indeed, the bits I did find made me think that perhaps you
> don't need defaultsite when you're using vhost--but I'm not even sure
> about that!

defaultsite= is there to 'fix' client requests which even today omit a
Host: header.
When vhost is used its optional but recommended.
When vhost is not used it might be required. Or it may not, if you only
have one peer and pass all requests there without checking the domain name
like this.

> 
> (The 2.7 change notes say that for http_port "Accelerator mode options
> cleaned up (accel, defaultsite, vport, vhost and combinations
> thereof)". Is the difference in behaviour here related to that
> clean-up?)

I would think so yes.

> 
> If anyone can explain--or point me to the explanation I missed--I'd
> appreciate it.
> 
> === squid.conf created by plone.recipe.squid
> 
> # This configuration file requires squid 2.6+.  It is untested with squid
> 3.x.

Notes about a 3.x upgrade inline as well...

> 
> visible_hostname [domain]
> http_port 127.0.0.1:10932 accel vhost defaultsite=[domain]
> pid_filename /home/[user]/webapps/plone/var/squid.pid
> 
> ## Log files (http://wiki.squid-cache.org/SquidFaq/SquidLogs)
> cache_access_log /home/[user]/webapps/plone/var/log/squid-access.log
> cache_log /home/[user]/webapps/plone/var/log/squid-cache.log
> cache_store_log none
> 
> # Cache storage
> cache_dir ufs /home/[user]/webapps/plone/var/squidstorage 1000 16 256
> cache_mem 64 MB
> maximum_object_size 10 MB
> maximum_object_size_in_memory 1 MB
> 
> # Purge access - zope servers can purge but nobody else
> # (works best if Zope and Apache are on different IPs)
> acl zope_servers src 127.0.0.1
> acl purge method PURGE
> http_access allow zope_servers purge
> http_access deny purge
> 
> # Deny caching of POST requests
> acl post_requests method POST
> cache deny post_requests
> 
> # Cache Peers
> cache_peer 127.0.0.1 parent 45086 0 no-query originserver login=PASS
> name=server_0
> 
> # Cache Peer Access
> acl all src 0.0.0.0/0.0.0.0

Please make that "acl all src all". And note that it needs to be removed
entirely for squid 3.x to work.

> 
> cache_peer_access server_0 allow all

Amos

Re: [squid-users] squid vport

2009-09-28 Thread Amos Jeffries 

Note: If Henrik corrects me on any of this believe him. I'm a little bit
fuzzy too.

On Tue, 29 Sep 2009 11:50:49 +0800, "wangwen"  wrote:
> Hi All:
> 
> The vport option Confused me for a long time.
> what is the meaning of "vport" option in the "http_port"?
> Can anyone give me an example to explain it, thank you.
> 
> I did a test.
> The Squid.conf as follows：
> http_port 192.168.0.164:88
> accel defaultsite=192.168.24.198
> acl all src 0.0.0.0/0.0.0.0
> http_access allow all
> cache_peer 192.168.24.198 parent 8088 0 no-query originserver
> 
> When Clients access http://192.168.0.164:88/rdims/index.jsp
> HTTP request header which Squid sent to backend server is:
> 
> GET /rdims/index.jsp HTTP/1.0
> Accept: */*
> Accept-Language: zh-cn
> UA-CPU: x86
> Accept-Encoding: gzip, deflate
> User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
.NET
> CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
> Host: 192.168.24.198
> Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
> X-Forwarded-For: 192.168.12.48
> Cache-Control: max-age=259200
> Connection: keep-alive
> 
> My originserver port is 8088,Why HOST is 192.168.0.164, I think that the
> HOST should be 92.168.24.198:8080.


"accel defaultsite=192.168.24.198" by itself means...

... 'accelerate' the URL.
... use 192.168.24.198 as the Host: header content if no domain name
present (in the URL).

Client sent squid:
GET /rdims/index.jsp HTTP/1.0
Host: 192.168.0.164:88


... /rdims/index.jsp contains no domain name, so the
defaultsite=192.168.24.198 set "Host: 192.168.24.198"

> 
> I alter HTTP_Port as follow:
> http_port 192.168.0.164:88 accel vhost defaultsite=192.168.24.198 vport
> When Clients access http://192.168.0.164:88/rdims/index.jsp
> HTTP request header which Squid sent to backend server is:
> 
> GET /rdims/index.jsp HTTP/1.0
> Accept: */*
> Accept-Language: zh-cn
> UA-CPU: x86
> Accept-Encoding: gzip, deflate
> User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
.NET
> CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
> Host: 192.168.24.198:88
> Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
> X-Forwarded-For: 192.168.12.48
> Cache-Control: max-age=259200
> Connection: keep-alive
> 
> Now why the Host is 192.168.0.164:88?

"accel vhost defaultsite=192.168.24.198 vport" means ...

... 'accelerate' the URL.
... (vhost) use client-given Host: header when available.
... (vport) use the client-given Host: HTTP port (default the peers
listening port if none found).
... (defaultsite=) use 192.168.24.198 if neither of the above give a usable
domain/IP.


The client sends:

Client sent squid:
GET /rdims/index.jsp HTTP/1.0
Host: 192.168.0.164:88


So Squid check the URL. Finds /rdims/index.jsp.
... checks the Host: header. Finds 192.168.0.164:88, making
URL=http://192.168.0.164:88/rdims/index.jsp
... checks the Host: header port. Finds 88, making
URL=http://192.168.0.164:88/rdims/index.jsp
... has a domain '192.168.0.164', so skips defaultsite

And passes on:
GET /rdims/index.jsp HTTP/1.0
Host: 192.168.0.164:88


> 
> I alter HTTP_Port again: http_port 192.168.0.164:88 accel vhost
> defaultsite=192.168.24.198 vport=8088
>  When Clients access http://192.168.0.164:88/rdims/index.jsp
>  HTTP request header which Squid sent to backend server is:
> 
> GET /rdims/index.jsp HTTP/1.0
> Accept: */*
> Accept-Language: zh-cn
> UA-CPU: x86
> Accept-Encoding: gzip, deflate
> User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1;
.NET
> CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
> Host: 192.168.24.198
> Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
> X-Forwarded-For: 192.168.12.48
> Cache-Control: max-age=259200
> Connection: keep-alive
> 
> Why the host is 192.168.0.164:88?

? its not.

> I think that the HOST should be
> 92.168.24.198:8088.

"accel vhost defaultsite=192.168.24.198 vport=8080" means ...

... 'accelerate' the URL.
... (vhost) use client-given Host: header when available.
... (vport=) use the client-given Host: HTTP port (default 8088 if none
found).
... (defaultsite=) use 192.168.24.198 if neither of the above give a usable
domain/IP.

So Squid check the URL. Finds /rdims/index.jsp.
... checks the Host: header. Finds 192.168.0.164:88, making
URL=http://192.168.0.164:88/rdims/index.jsp
... checks the Host: header port. Finds 88, making
URL=http://192.168.0.164:88/rdims/index.jsp
... has a domain '192.168.0.164', so skips defaultsite

I think it should have been: "Host: 192.168.0.164:88" maybe something went
wrong and caused defaultsite to be used. Henrik might know.

> At last the vport must be used with vhost?

I don't think so.

Amos

[squid-users] squid.conf and Squid 2.6 vs. Squid 2.7

2009-09-28 Thread Michael Lenaghan 
I used a Python recipe from PyPi to build, install and configure
Squid. The same recipe, config, etc. breaks when moving from Squid 2.6
to Squid 2.7. Other people have reported seeing the same error but no
one has bothered to track down a fix. I have one now, but I really
only got it by trying a bunch of different things. Before reporting
the fix back to various groups I'd like to understand what broke and
why the change made a difference.

In my particular case Squid was being used as a reverse (caching)
proxy in an Nginx -> Squid -> Zope chain. Squid was running on port
10932 and Zope on 45086. With 2.6 everything worked, but when
switching to 2.7 Squid's port number became visible in URLs. For
example, in the main page the href to the logo appeared as:

http://[domain]:10932/logo.jpg

rather than:

http://[domain]/logo.jpg

I've pasted the original conf file below (with domain names and such
abstracted out), but the fix involved the addition of vport=80 to the
http_port line:

http_port 127.0.0.1:10932 accel vhost vport=80 defaultsite=[domain]

I've had a very difficult time finding good docs for vhost, vport and
defaultsite. I've looked and I've searched in many places, but I
haven't found anything that would help me explain *why* this change
worked. Indeed, the bits I did find made me think that perhaps you
don't need defaultsite when you're using vhost--but I'm not even sure
about that!

(The 2.7 change notes say that for http_port "Accelerator mode options
cleaned up (accel, defaultsite, vport, vhost and combinations
thereof)". Is the difference in behaviour here related to that
clean-up?)

If anyone can explain--or point me to the explanation I missed--I'd
appreciate it.

=== squid.conf created by plone.recipe.squid

# This configuration file requires squid 2.6+.  It is untested with squid 3.x.

visible_hostname [domain]
http_port 127.0.0.1:10932 accel vhost defaultsite=[domain]
pid_filename /home/[user]/webapps/plone/var/squid.pid

## Log files (http://wiki.squid-cache.org/SquidFaq/SquidLogs)
cache_access_log /home/[user]/webapps/plone/var/log/squid-access.log
cache_log /home/[user]/webapps/plone/var/log/squid-cache.log
cache_store_log none

# Cache storage
cache_dir ufs /home/[user]/webapps/plone/var/squidstorage 1000 16 256
cache_mem 64 MB
maximum_object_size 10 MB
maximum_object_size_in_memory 1 MB

# Purge access - zope servers can purge but nobody else
# (works best if Zope and Apache are on different IPs)
acl zope_servers src 127.0.0.1
acl purge method PURGE
http_access allow zope_servers purge
http_access deny purge

# Deny caching of POST requests
acl post_requests method POST
cache deny post_requests

# Cache Peers
cache_peer 127.0.0.1 parent 45086 0 no-query originserver login=PASS
name=server_0

# Cache Peer Access
acl all src 0.0.0.0/0.0.0.0

cache_peer_access server_0 allow all

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Chris Woodfield 
Barring the development of an in-code fix, I think your best bet is to  
take the config generator route you describe, but then run the  
generator from your squid initscript triggered by "start" or "reload".  
Beyond pushing out an updated text file to your boxes and building the  
cache_peer lines from there (excluding one's own IP), some other  
solutions to get that list could be:


- Get list of cache_peer IPs from the A responses to a DNS hostname  
query
- If using multicast ICP/HTCP, script a query to the configured  
multicast address, and add the IPs of the responding hosts to the  
cache_peer config
- Put a text file on an internal web server that contains the list of  
peer IPs


Another solution could be to use a multi-level CARP config, which  
incidentally scales far better horizontally than ICP/HTCP, as it  
eliminates the iterative "sideways" queries altogether by hashing URLs  
to parent cache_peers. In this setup, you'd run two squids on each box  
- one "edge" squid that answers client queries but does no caching on  
its own, and a "parent" squid that listens on a different IP or TCP  
port that actually does the caching. This solves your issue by giving  
every edge instance the same list of parent cache_peers - it just so  
happens that one of them is an IP/port that happens to be local, but  
not the same instance. Likewise, all the parent instances can have  
identical configs.


That said, it is a bit of work to get multiple squids running on a  
single box - you need your initscripts to call squid with different -f  
options for the different configs, set up separate log files, etc.


-C

On Sep 28, 2009, at 8:24 PM, Chris Hostetter wrote:



: The DNS way would indeed be nice. It's not possible in current Squid
: however, if anyone is able to sponsor some work it might be doable.

If i can demonstrate enough advantages in getting peering to work i  
might
just be able to convince someone to think about doing that ... but  
that

also assumes i can get the operations team adament enough to protest
having a hack where they need to run a "config_generator" script on
every box whenever a cluster changes (because a script like that  
would be

fairly straight forward to write as a one off, it's just harder to
implement as a general purpose feature in squid)

: With Squid-2.7 you can use the 'include' directive to split the  
squid.conf
: apart and contain the unique per-machine parts in a separate file  
to the

: shared parts.

yeah, i'm already familiar with inlcude, but either way i need a
per-machine snippetto get arround the "sibling to self" problem  
*and* a
way to reconfig when the snippet changes (because of the cluster  
changing

problem)

-Hoss

[squid-users] squid vport

2009-09-28 Thread wangwen 

Hi All:

The vport option Confused me for a long time.
what is the meaning of "vport" option in the "http_port"?
Can anyone give me an example to explain it, thank you.

I did a test.
The Squid.conf as follows：
http_port 192.168.0.164:88
accel defaultsite=192.168.24.198
acl all src 0.0.0.0/0.0.0.0
http_access allow all
cache_peer 192.168.24.198 parent 8088 0 no-query originserver

When Clients access http://192.168.0.164:88/rdims/index.jsp
HTTP request header which Squid sent to backend server is:

GET /rdims/index.jsp HTTP/1.0
Accept: */*
Accept-Language: zh-cn
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
Host: 192.168.24.198
Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
X-Forwarded-For: 192.168.12.48
Cache-Control: max-age=259200
Connection: keep-alive

My originserver port is 8088,Why HOST is 192.168.0.164, I think that the
HOST should be 92.168.24.198:8080.

I alter HTTP_Port as follow:
http_port 192.168.0.164:88 accel vhost defaultsite=192.168.24.198 vport
When Clients access http://192.168.0.164:88/rdims/index.jsp
HTTP request header which Squid sent to backend server is:

GET /rdims/index.jsp HTTP/1.0
Accept: */*
Accept-Language: zh-cn
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
Host: 192.168.24.198:88
Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
X-Forwarded-For: 192.168.12.48
Cache-Control: max-age=259200
Connection: keep-alive

Now why the Host is 192.168.0.164:88?

I alter HTTP_Port again: http_port 192.168.0.164:88 accel vhost
defaultsite=192.168.24.198 vport=8088
When Clients access http://192.168.0.164:88/rdims/index.jsp
HTTP request header which Squid sent to backend server is:

GET /rdims/index.jsp HTTP/1.0
Accept: */*
Accept-Language: zh-cn
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET
CLR 2.0.50727; InfoPath.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)
Host: 192.168.24.198
Via: 1.1 szrd.com:88 (squid/2.6.STABLE21)
X-Forwarded-For: 192.168.12.48
Cache-Control: max-age=259200
Connection: keep-alive

Why the host is 192.168.0.164:88? I think that the HOST should be
92.168.24.198:8088.
At last the vport must be used with vhost?

Thank you.

Re: [squid-users] transparent integration with proxy on router

2009-09-28 Thread Todd Nine 

Hi Amos,
 If some of these entries are incorrect other than the redirect rule, 
it's unfortunately a bug with the pfSense module.  pfSense uses a nice 
little web GUI to create the squid.conf.  I don't have the ability to 
edit anything you see below except the rules I've defined are created 
automatically by pfsense.  If I overwrite it, I'll lose my changes when 
it restarts.  I'll take a look at the code as it's written in PHP and 
submit a feature request with a patch to clean up a lot of the 
boilerplate configuration.  Thanks for all the help, you've been a great 
resource to get this working quickly.


Todd

Amos Jeffries wrote:

On Tue, 29 Sep 2009 13:22:58 +1300, Todd Nine 
wrote:
  

Hi Amos,
  Here is my squid.conf.  I've just used the defaults and added a single 
rule.  We're pushing a lot of throughput (several gigs a day).  I've 
disabled writing to disk as we actually run from a USB appliance, and 
set the cache size to 1 GB (1024M) of RAM.  My main use of squid is not 
caching, but rather http redirection to save us money on our usage fees 
from our ISPs.



In which case you probably do want to look at caching. Since that can have
between 20% and 40% reduction in HTTP traffic going to your ISPs.


What version of squid is this?

  

Thanks again for the help!

File:
# Do not edit manually !
http_port 10.0.1.1:3128
http_port 10.0.2.1:3128
http_port 127.0.0.1:80 transparent
icp_port 0

pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_directory /usr/local/etc/squid/errors/English
icon_directory /usr/local/etc/squid/icons
visible_hostname router
cache_mgr ad...@localhost
access_log /var/squid/log/access.log
cache_log /var/squid/log/cache.log
cache_store_log none
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  10.0.1.0/255.255.255.0 10.0.2.0/255.255.255.0



Please use CIDR masks:
acl localnet src  10.0.1.0/24 10.0.2.0/24

  

uri_whitespace strip

cache_dir aufs /var/squid/cache 100 16 256



Huh? you said you disabled writing to disk. That above is using a 100MB
cache on the disk.

To disable disk caching use the 'cache_dir null ' storage type in Squid
older than 3.1, or remove all cache_dir from Squid-3.1+.

  

cache_mem 1024 MB
maximum_object_size 4 KB



1GB worth of 4KB objects is  a LOT of objects. If you have the memory to
spare 1GB for caching its probably best to allow moderate sized objects to
be cached in RAM. Setting Max size to 1MB should do. Though depending on
the popularity of video sites with your users they may also benefit from
10MB max object size (video causes a bump at 2MB-8MB apparently).

  

minimum_object_size 0 KB
cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
offline_mode off
dns_children 32



The above is only relevant with the obsolete 'dnsserver' helper. If you are
still using that you would get a great deal of performance boots by
changing to the internal DNS (requires a recompile).

  

cache_swap_low 90
cache_swap_high 95
acl donotcache dstdomain "/var/squid/acl/donotcache.acl"
cache deny donotcache
# No redirector configured



# Setup some default acls
acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255



Some more CIDR benefits:
acl all src all
acl localhost src 127.0.0.1

  
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  
3128 1025-65535

acl sslports port 443 563 
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic



More speed boost by dropping the above QUERY

  

http_access allow manager localhost
 
http_access deny manager

http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
reply_body_max_size 0 allow all
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow all

# Allow local network(s) on interface(s)
http_access allow localnet
# Custom options
#Set up our ACL for high throughput sites
acl high_throughput dstdomain .amazonaws.com .rapidshare.com
#Bind high throughput to the wireless interface
tcp_outgoing_address 116.90.140.xx high_throughput

# Default block all to be sure
http_access deny all



Amos Jeffries wrote:


On Tue, 29 Sep 2009 09:32:49 +1300, Todd Nine 
wrote:
  
  

Thanks for the help!  I read over the rules and it was quite easy to


set
  
up what I needed once I had the right directive.  I simply set up the 
following.


#Set up our ACL for high throughput sites
acl high_throughput dstdomain .amazonaws.com

#Bind high throughput to the wireless interface
tcp_outgoing_address 116.90.140.xx high_throughput

However we're having a side effect issue.  Our router box is a bit old 
(an old P4), and we can't keep up with the squid demands

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Amos Jeffries 
On Mon, 28 Sep 2009 17:57:02 -0700, George Herbert
 wrote:
> On Mon, Sep 28, 2009 at 5:24 PM, Chris Hostetter
>  wrote:
>>
>> : The DNS way would indeed be nice. It's not possible in current Squid
>> : however, if anyone is able to sponsor some work it might be doable.
>>
>> If i can demonstrate enough advantages in getting peering to work i
might
>> just be able to convince someone to think about doing that ... but that
>> also assumes i can get the operations team adament enough to protest
>> having a hack where they need to run a "config_generator" script on
>> every box whenever a cluster changes (because a script like that would
be
>> fairly straight forward to write as a one off, it's just harder to
>> implement as a general purpose feature in squid)
>>
>> : With Squid-2.7 you can use the 'include' directive to split the
>> squid.conf
>> : apart and contain the unique per-machine parts in a separate file to
>> the
>> : shared parts.
>>
>> yeah, i'm already familiar with inlcude, but either way i need a
>> per-machine snippetto get arround the "sibling to self" problem *and* a
>> way to reconfig when the snippet changes (because of the cluster
changing
>> problem)
>>
>> -Hoss
> 
> 
> What would be really nice is a command line option and a bit of code
> in the cache peer setup that recognizes own IP and ignores the entry,
> to make this problem just all go away...
> 
> I should code that up, but not early tonight...

If you do I'm happy to mentor the patch through submission and auditing.

Also related is the Via: and X-Forwarded-For header behavior.  Using HTCP
between the peers instead of ICP causes the full HTTP headers to be sent as
part of the sibling cache query. That allows peers to respond 'not me' or
something if they see their unique_hostname or visible_hostname in the Via:
header. Or their own IP in the X-Forwarded-For header.  I'm not sure what
the current Squid behavior is on that though, I think it has a good chance
at being there.

Amos

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread George Herbert 
On Mon, Sep 28, 2009 at 5:24 PM, Chris Hostetter
 wrote:
>
> : The DNS way would indeed be nice. It's not possible in current Squid
> : however, if anyone is able to sponsor some work it might be doable.
>
> If i can demonstrate enough advantages in getting peering to work i might
> just be able to convince someone to think about doing that ... but that
> also assumes i can get the operations team adament enough to protest
> having a hack where they need to run a "config_generator" script on
> every box whenever a cluster changes (because a script like that would be
> fairly straight forward to write as a one off, it's just harder to
> implement as a general purpose feature in squid)
>
> : With Squid-2.7 you can use the 'include' directive to split the squid.conf
> : apart and contain the unique per-machine parts in a separate file to the
> : shared parts.
>
> yeah, i'm already familiar with inlcude, but either way i need a
> per-machine snippetto get arround the "sibling to self" problem *and* a
> way to reconfig when the snippet changes (because of the cluster changing
> problem)
>
> -Hoss


What would be really nice is a command line option and a bit of code
in the cache peer setup that recognizes own IP and ignores the entry,
to make this problem just all go away...

I should code that up, but not early tonight...


-- 
-george william herbert
george.herb...@gmail.com

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Amos Jeffries 
On Mon, 28 Sep 2009 17:24:16 -0700 (PDT), Chris Hostetter
 wrote:
> : The DNS way would indeed be nice. It's not possible in current Squid
> : however, if anyone is able to sponsor some work it might be doable.
> 
> If i can demonstrate enough advantages in getting peering to work i might

> just be able to convince someone to think about doing that ... but that 
> also assumes i can get the operations team adament enough to protest 
> having a hack where they need to run a "config_generator" script on
> every box whenever a cluster changes (because a script like that would be

> fairly straight forward to write as a one off, it's just harder to 
> implement as a general purpose feature in squid)
> 
> : With Squid-2.7 you can use the 'include' directive to split the
> squid.conf
> : apart and contain the unique per-machine parts in a separate file to
the
> : shared parts.
> 
> yeah, i'm already familiar with inlcude, but either way i need a 
> per-machine snippetto get arround the "sibling to self" problem *and* a 
> way to reconfig when the snippet changes (because of the cluster changing

> problem)
> 
> -Hoss

For a bit of leverage; every time the config changes squid needs to be
reconfigured, which causes a short outage.

I've been thinking about it, and the multiple-IPs for a cache_peer might
work when a DNS name is entered as first parameter of cache_peer. But only
the IPs detected at startup/reconfigure time will be used so you don't
really evade the reconfigure outage with that either.

Amos

Re: [squid-users] transparent integration with proxy on router

2009-09-28 Thread Amos Jeffries 

Oops, I hit send by mistake before I finished. Here is the rest...

On Tue, 29 Sep 2009 13:22:58 +1300, Todd Nine 
wrote:
> Hi Amos,
>   Here is my squid.conf.  I've just used the defaults and added a single 
> rule.  We're pushing a lot of throughput (several gigs a day).  I've 
> disabled writing to disk as we actually run from a USB appliance, and 
> set the cache size to 1 GB (1024M) of RAM.  My main use of squid is not 
> caching, but rather http redirection to save us money on our usage fees 
> from our ISPs.
> 
> Thanks again for the help!
> 
> File:

> 
> # Setup some default acls
> acl all src 0.0.0.0/0.0.0.0
> acl localhost src 127.0.0.1/255.255.255.255
> acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  
> 3128 1025-65535
> acl sslports port 443 563 
> acl manager proto cache_object
> acl purge method PURGE
> acl connect method CONNECT
> acl dynamic urlpath_regex cgi-bin \?
> cache deny dynamic

bit of a speed boost dropping the above two QUERY lines and adding:
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

as the pattern directly above the . (dot) refresh_pattern.

> http_access allow manager localhost
>  
> http_access deny manager
> http_access allow purge localhost
> http_access deny purge
> http_access deny !safeports
> http_access deny CONNECT !sslports
> 
> # Always allow localhost connections
> http_access allow localhost
> 
> request_body_max_size 0 KB
> reply_body_max_size 0 allow all
> delay_pools 1
> delay_class 1 2
> delay_parameters 1 -1/-1 -1/-1
> delay_initial_bucket_level 100
> delay_access 1 allow all

Huh? This is adding a lot of useless work to Squid.

-1/-1 is an 'unlimited' pool. The above configuration is identical in
effect as not having delay pools at all.

> 
> # Allow local network(s) on interface(s)
> http_access allow localnet
> # Custom options
> #Set up our ACL for high throughput sites
> acl high_throughput dstdomain .amazonaws.com .rapidshare.com
> #Bind high throughput to the wireless interface
> tcp_outgoing_address 116.90.140.xx high_throughput
> 
> # Default block all to be sure
> http_access deny all
> 

the end.

Amos

> 
> 
> Amos Jeffries wrote:
>> On Tue, 29 Sep 2009 09:32:49 +1300, Todd Nine 
>> wrote:
>>   
>>> Thanks for the help!  I read over the rules and it was quite easy to
set
>>>
>>> up what I needed once I had the right directive.  I simply set up the 
>>> following.
>>>
>>> #Set up our ACL for high throughput sites
>>> acl high_throughput dstdomain .amazonaws.com
>>>
>>> #Bind high throughput to the wireless interface
>>> tcp_outgoing_address 116.90.140.xx high_throughput
>>>
>>> However we're having a side effect issue.  Our router box is a bit old 
>>> (an old P4), and we can't keep up with the squid demands due to the 
>>> number of users with 2 GB of ram.  Is there a directive that I can tell

>>> squid not to proxy connections unless they meet the "high_throughput" 
>>> acl?  I looked and couldn't find any bypass directives that met what I 
>>> needed.
>>>
>>> Thanks,
>>> Todd
>>> 
>>
>> Once connections have already entered Squid its too late to not send
them
>> to Squid.
>>
>> I have run Squid on P4s routers with 256MB of RAM for hundreds of
domains
>> and dozens of clients without having the box run up much of a sweat.
What
>> is your load (both CPU box load, and visitor rates, bandwidth) like?
>> Also check your other configuration and access controls are using
>> efficient
>> methods, if you don't know what those are already I'm happy to give
>> configs
>> an audit and point things that need adjusting out.
>>
>> Amos
>>
>>   
>>> Amos Jeffries wrote:
>>> 
 On Mon, 28 Sep 2009 16:21:16 +1300, Todd Nine

 wrote:
   
   
> Hi all,
>   I'm using squid on a pfSense router we've built.  We have 2 
> connections, one we pay for usage (DSL) and one we do not (Wireless).

>
> We use Amazon S3 extensively at work.  We've been attempting to route

> all traffic over the wireless via an IP range, but as S3 can change
> 
>> IPs,
>>   
> this doesn't work and we end up with a large bill for our DSL.  Is it

> possible to have squid route connections via a specific interface if
a
>
> hostname such as "amazonaws.com" is in the HTTP request header?
>
> Thanks,
> Todd
> 
> 
 Yes you can.

 Find an IP assigned to the interface you want traffic to go out. Use
 the
 tcp_outgoing_addr directive and ACLs that match the requests to make
   
>> sure
>>   
 all the requests to that domain are assigned that outgoing address. 
   
>> Then
>>   
 make sure the OS sends traffic from that IP out the right interface.

 Amos

Re: [squid-users] not caching enough

2009-09-28 Thread Chris Hostetter 

: > Ok, it has come to my attention that /cache only grwos when I run a
: report
...
: > cache_dir ufs /cache 50 256 256

...if i can start by asking a silly question: i assume your cache is 
not yet at the maximum size?

The next silly question: have you done any analysis of the squid logs 
and the types of requests being made through your cache?  I know nothing 
about your usecase but perhaps you've just reached a steady state where 
everything that can be cached is already being overwritten when it 
expires.

If nothing else a simple grep of your access.log for "_HIT/200 " should
answer your main question...

: > using sarg. So it maybe that my proxy server is working, but only as a
: > proxy
: > and not as a web cache proxy. Below is my squid.conf file with the

-Hoss

Re: [squid-users] transparent integration with proxy on router

2009-09-28 Thread Amos Jeffries 
On Tue, 29 Sep 2009 13:22:58 +1300, Todd Nine 
wrote:
> Hi Amos,
>   Here is my squid.conf.  I've just used the defaults and added a single 
> rule.  We're pushing a lot of throughput (several gigs a day).  I've 
> disabled writing to disk as we actually run from a USB appliance, and 
> set the cache size to 1 GB (1024M) of RAM.  My main use of squid is not 
> caching, but rather http redirection to save us money on our usage fees 
> from our ISPs.

In which case you probably do want to look at caching. Since that can have
between 20% and 40% reduction in HTTP traffic going to your ISPs.


What version of squid is this?

> 
> Thanks again for the help!
> 
> File:
> # Do not edit manually !
> http_port 10.0.1.1:3128
> http_port 10.0.2.1:3128
> http_port 127.0.0.1:80 transparent
> icp_port 0
> 
> pid_filename /var/run/squid.pid
> cache_effective_user proxy
> cache_effective_group proxy
> error_directory /usr/local/etc/squid/errors/English
> icon_directory /usr/local/etc/squid/icons
> visible_hostname router
> cache_mgr ad...@localhost
> access_log /var/squid/log/access.log
> cache_log /var/squid/log/cache.log
> cache_store_log none
> shutdown_lifetime 3 seconds
> # Allow local network(s) on interface(s)
> acl localnet src  10.0.1.0/255.255.255.0 10.0.2.0/255.255.255.0

Please use CIDR masks:
acl localnet src  10.0.1.0/24 10.0.2.0/24

> uri_whitespace strip
> 
> cache_dir aufs /var/squid/cache 100 16 256

Huh? you said you disabled writing to disk. That above is using a 100MB
cache on the disk.

To disable disk caching use the 'cache_dir null ' storage type in Squid
older than 3.1, or remove all cache_dir from Squid-3.1+.

> cache_mem 1024 MB
> maximum_object_size 4 KB

1GB worth of 4KB objects is  a LOT of objects. If you have the memory to
spare 1GB for caching its probably best to allow moderate sized objects to
be cached in RAM. Setting Max size to 1MB should do. Though depending on
the popularity of video sites with your users they may also benefit from
10MB max object size (video causes a bump at 2MB-8MB apparently).

> minimum_object_size 0 KB
> cache_replacement_policy heap LFUDA
> memory_replacement_policy heap GDSF
> offline_mode off
> dns_children 32

The above is only relevant with the obsolete 'dnsserver' helper. If you are
still using that you would get a great deal of performance boots by
changing to the internal DNS (requires a recompile).

> cache_swap_low 90
> cache_swap_high 95
> acl donotcache dstdomain "/var/squid/acl/donotcache.acl"
> cache deny donotcache
> # No redirector configured
> 
> 
> 
> # Setup some default acls
> acl all src 0.0.0.0/0.0.0.0
> acl localhost src 127.0.0.1/255.255.255.255

Some more CIDR benefits:
acl all src all
acl localhost src 127.0.0.1

> acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  
> 3128 1025-65535
> acl sslports port 443 563 
> acl manager proto cache_object
> acl purge method PURGE
> acl connect method CONNECT
> acl dynamic urlpath_regex cgi-bin \?
> cache deny dynamic

More speed boost by dropping the above QUERY

> http_access allow manager localhost
>  
> http_access deny manager
> http_access allow purge localhost
> http_access deny purge
> http_access deny !safeports
> http_access deny CONNECT !sslports
> 
> # Always allow localhost connections
> http_access allow localhost
> 
> request_body_max_size 0 KB
> reply_body_max_size 0 allow all
> delay_pools 1
> delay_class 1 2
> delay_parameters 1 -1/-1 -1/-1
> delay_initial_bucket_level 100
> delay_access 1 allow all
> 
> # Allow local network(s) on interface(s)
> http_access allow localnet
> # Custom options
> #Set up our ACL for high throughput sites
> acl high_throughput dstdomain .amazonaws.com .rapidshare.com
> #Bind high throughput to the wireless interface
> tcp_outgoing_address 116.90.140.xx high_throughput
> 
> # Default block all to be sure
> http_access deny all
> 
> 
> 
> Amos Jeffries wrote:
>> On Tue, 29 Sep 2009 09:32:49 +1300, Todd Nine 
>> wrote:
>>   
>>> Thanks for the help!  I read over the rules and it was quite easy to
set
>>>
>>> up what I needed once I had the right directive.  I simply set up the 
>>> following.
>>>
>>> #Set up our ACL for high throughput sites
>>> acl high_throughput dstdomain .amazonaws.com
>>>
>>> #Bind high throughput to the wireless interface
>>> tcp_outgoing_address 116.90.140.xx high_throughput
>>>
>>> However we're having a side effect issue.  Our router box is a bit old 
>>> (an old P4), and we can't keep up with the squid demands due to the 
>>> number of users with 2 GB of ram.  Is there a directive that I can tell

>>> squid not to proxy connections unless they meet the "high_throughput" 
>>> acl?  I looked and couldn't find any bypass directives that met what I 
>>> needed.
>>>
>>> Thanks,
>>> Todd
>>> 
>>
>> Once connections have already entered Squid its too late to not send
them
>> to Squid.
>>
>> I have run Squid on P4s routers with 256MB of RAM for hundreds of
domains
>> and dozens of clients without having the b

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Chris Hostetter 

: The DNS way would indeed be nice. It's not possible in current Squid
: however, if anyone is able to sponsor some work it might be doable.

If i can demonstrate enough advantages in getting peering to work i might 
just be able to convince someone to think about doing that ... but that 
also assumes i can get the operations team adament enough to protest 
having a hack where they need to run a "config_generator" script on
every box whenever a cluster changes (because a script like that would be 
fairly straight forward to write as a one off, it's just harder to 
implement as a general purpose feature in squid)

: With Squid-2.7 you can use the 'include' directive to split the squid.conf
: apart and contain the unique per-machine parts in a separate file to the
: shared parts.

yeah, i'm already familiar with inlcude, but either way i need a 
per-machine snippetto get arround the "sibling to self" problem *and* a 
way to reconfig when the snippet changes (because of the cluster changing 
problem)

-Hoss

Re: [squid-users] not caching enough

2009-09-28 Thread Amos Jeffries 
On Mon, 28 Sep 2009 06:36:33 -0700 (PDT), ant2ne 
wrote:
> Ok, it has come to my attention that /cache only grwos when I run a
report
> using sarg. So it maybe that my proxy server is working, but only as a
> proxy
> and not as a web cache proxy. Below is my squid.conf file with the
comment
> #
> invert grepped out. Please review and tell me what it is that I need to
> change to turn this proxy server into a web cache server.
> 
> below is me squid.conf
> acl all src 0.0.0.0/0.0.0.0

Please use:
acl all src all

> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8

Please instead use:
acl localhost src 127.0.0.1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/8

> acl purge method PURGE
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access deny manager
> http_access allow purge localhost
> http_access deny purge
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny to_localhost
> acl our_networks src 10.60.140.0/24
> http_access allow our_networks
> http_access allow localhost
> http_access allow all

So everybody on the entire Internet is allowed to use your cache? not good.

> http_access deny all
> icp_access allow all

Also not good, for the same reasons. Try:

icp_access allow our_networks
icp_access allow localhost
icp_access deny all


> http_port 3128
> hierarchy_stoplist cgi-bin ?
> cache_dir ufs /cache 50 256 256
> maximum_object_size 32768 KB

With sich large disk I would not worry about limiting objects to small
sizes. You probably want set that as:
maximum_object_size 1 GB

... or "0 KB " for 'off'

> access_log /var/log/squid/access.log squid

> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY

There is a sizable amount of dynamic content which is cacheable these days.
Erase the above two lines.

> refresh_pattern ^ftp: 144020% 10080
> refresh_pattern ^gopher:  14400%  1440

Add at this exact spot in the config:
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

> refresh_pattern . 0   20% 4320
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> extension_methods REPORT MERGE MKACTIVITY CHECKOUT
> hosts_file /etc/hosts
> coredump_dir /var/spool/squid
> visible_hostname AHSPX01

Amos

Re: [squid-users] transparent integration with proxy on router

2009-09-28 Thread Amos Jeffries 
On Tue, 29 Sep 2009 09:32:49 +1300, Todd Nine 
wrote:
> Thanks for the help!  I read over the rules and it was quite easy to set 
> up what I needed once I had the right directive.  I simply set up the 
> following.
> 
> #Set up our ACL for high throughput sites
> acl high_throughput dstdomain .amazonaws.com
> 
> #Bind high throughput to the wireless interface
> tcp_outgoing_address 116.90.140.xx high_throughput
> 
> However we're having a side effect issue.  Our router box is a bit old 
> (an old P4), and we can't keep up with the squid demands due to the 
> number of users with 2 GB of ram.  Is there a directive that I can tell 
> squid not to proxy connections unless they meet the "high_throughput" 
> acl?  I looked and couldn't find any bypass directives that met what I 
> needed.
> 
> Thanks,
> Todd

Once connections have already entered Squid its too late to not send them
to Squid.

I have run Squid on P4s routers with 256MB of RAM for hundreds of domains
and dozens of clients without having the box run up much of a sweat. What
is your load (both CPU box load, and visitor rates, bandwidth) like?
Also check your other configuration and access controls are using efficient
methods, if you don't know what those are already I'm happy to give configs
an audit and point things that need adjusting out.

Amos

> 
> Amos Jeffries wrote:
>> On Mon, 28 Sep 2009 16:21:16 +1300, Todd Nine 
>> wrote:
>>   
>>> Hi all,
>>>   I'm using squid on a pfSense router we've built.  We have 2 
>>> connections, one we pay for usage (DSL) and one we do not (Wireless).  
>>> We use Amazon S3 extensively at work.  We've been attempting to route 
>>> all traffic over the wireless via an IP range, but as S3 can change
IPs,
>>>
>>> this doesn't work and we end up with a large bill for our DSL.  Is it 
>>> possible to have squid route connections via a specific interface if a 
>>> hostname such as "amazonaws.com" is in the HTTP request header?
>>>
>>> Thanks,
>>> Todd
>>> 
>>
>> Yes you can.
>>
>> Find an IP assigned to the interface you want traffic to go out. Use the
>> tcp_outgoing_addr directive and ACLs that match the requests to make
sure
>> all the requests to that domain are assigned that outgoing address. 
Then
>> make sure the OS sends traffic from that IP out the right interface.
>>
>> Amos
>>

Re: [squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Amos Jeffries 
On Mon, 28 Sep 2009 15:04:35 -0700 (PDT), Chris Hostetter
 wrote:
> Background Information...
> 
> My company currently runs several "clusters" of application servers
behind 
> load balancers, which are each in turn sitting behind a "cluster" of
squid 
> machines configured as accelerators. each squid cluster is then sitting 
> behind a load balancer that is hit by our clients.
> 
> To elaborate: The hostname appAA resolves to a load balancer which
proxies 
> to appAA-squid1, appAA-squid2, appAA-squid2, etc...  Each of the 
> appAA-squidX machines is configured as a standalone accelerator (using 
> cache_peer ... parent no-query originserver) for appAA-backend. 
> appAA-backend resolves to a load balancer which proxies to
appAA-backend1, 
> appAA-backend2, appAA-backend3, etc...  Likewise for appBB, appCC, appDD,

> etc...
> 
> None of these squid instances know anything about each other.  in the
case 
> of appAA-squidX vs appBB-squidX this is a good thing, because the entire 
> point of isolating these apps is for QoS commitments, and ensuring that 
> heavy load or catastrophic failure on one app doesn't affect another app.
> 
> In the case of appAA-squidX vs appAA-squidY it definitely seems like
cache 
> peering would be advantageous here.
> 
> 
> The Problem(s)...
> 
> Our operations team is pretty adamant about software/configs deployed to 
> boxes in a clustering needing to be the same for every box in the
cluster. 
> The goal is understandable: they don't want to need custom install steps 
> for every individual machine.  So while my dev setup of a 5 machine squid

> cluster each with 4 distinct "cache_peer ... sibling" lines works great
so 
> far, i can't deploy a unique squid.conf for each machine in a cluster.
> 
> I could probably put a hack into our build system to check the current 
> hostname at installation time and remove any cache_peer lines refering to

> that hostname -- but before i jumped through those hoops i wanted to 
> sanity check that there wasn't an easier way to do this in squid.
> 
> is there any easy way to reuse the same cache_peer config options on 
> multiple instances, but keep squid smart enough that it doesn't bother 
> trying to peer with itself?
> 
> (I had a glimmer of an idea about using ACL rules for this, but didn't 
> work it through all the way because it seemed like at best that would 
> cause squid to deny requests from itself, not prevent it from attempting 
> the request in the first place)
> 
> I have a hard time imaging that i'm the first person to have this
problem, 
> but i couldn't find any obvious solutions in the mail archives.
> 
> 
> A slightly bigger problem is what to do when the cluster changes, either 
> because a machine is removed for maintenance issues or because a machine 
> is added due to because of increases in load.  In our current setup this 
> is a no-brainer: tell the load balancer when you add/remove a machine and

> everything just works -- none of the boxes know anything about each
other, 
> and they all run identical configs.
> 
> In order to setup sibling peering, it seems like i would need to 
> deploy/reload configs (with an updated list of cache_peer directives) to 
> every machine in the cluster anytime a new box is added or removed in 
> order for all of the siblings to know about each other.
> 
> Is there an easier way to coordinate the "discovery" of new siblings 
> automatically?
> 
> An ideal situation would be to use a DNS hostname in the cache_peer line 
> that resolves to multiple IPs and have squid re-resolve the hostname 
> periodically and update the list of peers based on *all* the addresses 
> associated with that name -- but from what i can tell squid will just 
> picking a single address (DNS Round-Robin style).
> 
> 
> Any advice or suggestions for managing peers like this would be 
> appreciated.

The DNS way would indeed be nice. It's not possible in current Squid
however, if anyone is able to sponsor some work it might be doable.

With Squid-2.7 you can use the 'include' directive to split the squid.conf
apart and contain the unique per-machine parts in a separate file to the
shared parts.

Amos

[squid-users] Managing clusters of siblings (squid2.7)

2009-09-28 Thread Chris Hostetter 


Background Information...

My company currently runs several "clusters" of application servers behind 
load balancers, which are each in turn sitting behind a "cluster" of squid 
machines configured as accelerators. each squid cluster is then sitting 
behind a load balancer that is hit by our clients.


To elaborate: The hostname appAA resolves to a load balancer which proxies 
to appAA-squid1, appAA-squid2, appAA-squid2, etc...  Each of the 
appAA-squidX machines is configured as a standalone accelerator (using 
cache_peer ... parent no-query originserver) for appAA-backend. 
appAA-backend resolves to a load balancer which proxies to appAA-backend1, 
appAA-backend2, appAA-backend3, etc...  Likewise for appBB, appCC, appDD, 
etc...


None of these squid instances know anything about each other.  in the case 
of appAA-squidX vs appBB-squidX this is a good thing, because the entire 
point of isolating these apps is for QoS commitments, and ensuring that 
heavy load or catastrophic failure on one app doesn't affect another app.


In the case of appAA-squidX vs appAA-squidY it definitely seems like cache 
peering would be advantageous here.



The Problem(s)...

Our operations team is pretty adamant about software/configs deployed to 
boxes in a clustering needing to be the same for every box in the cluster. 
The goal is understandable: they don't want to need custom install steps 
for every individual machine.  So while my dev setup of a 5 machine squid 
cluster each with 4 distinct "cache_peer ... sibling" lines works great so 
far, i can't deploy a unique squid.conf for each machine in a cluster.


I could probably put a hack into our build system to check the current 
hostname at installation time and remove any cache_peer lines refering to 
that hostname -- but before i jumped through those hoops i wanted to 
sanity check that there wasn't an easier way to do this in squid.


is there any easy way to reuse the same cache_peer config options on 
multiple instances, but keep squid smart enough that it doesn't bother 
trying to peer with itself?


(I had a glimmer of an idea about using ACL rules for this, but didn't 
work it through all the way because it seemed like at best that would 
cause squid to deny requests from itself, not prevent it from attempting 
the request in the first place)


I have a hard time imaging that i'm the first person to have this problem, 
but i couldn't find any obvious solutions in the mail archives.



A slightly bigger problem is what to do when the cluster changes, either 
because a machine is removed for maintenance issues or because a machine 
is added due to because of increases in load.  In our current setup this 
is a no-brainer: tell the load balancer when you add/remove a machine and 
everything just works -- none of the boxes know anything about each other, 
and they all run identical configs.


In order to setup sibling peering, it seems like i would need to 
deploy/reload configs (with an updated list of cache_peer directives) to 
every machine in the cluster anytime a new box is added or removed in 
order for all of the siblings to know about each other.


Is there an easier way to coordinate the "discovery" of new siblings 
automatically?


An ideal situation would be to use a DNS hostname in the cache_peer line 
that resolves to multiple IPs and have squid re-resolve the hostname 
periodically and update the list of peers based on *all* the addresses 
associated with that name -- but from what i can tell squid will just 
picking a single address (DNS Round-Robin style).



Any advice or suggestions for managing peers like this would be 
appreciated.



-Hoss

[squid-users] range_offset_limit per domain

2009-09-28 Thread Matthew Morgan 
Is it possible to set range_offset_limit per domain?  This really helps 
when caching windows updates; however, if I turn it on, my bandwidth 
gets eaten up any time one of my client machines requests part of a file 
on a CDN (like the first frame of an online video).

[squid-users] Re: Re: Re: Re: Re: squid_kerb_auth.... Key Version number?

2009-09-28 Thread Markus Moeller 


"Mrvka Andreas"  wrote in message 
news:200909281546.30273@tuv.at...

Hello Markus,

I thought there will be more changes in wiki than what you have written.

You write about either use msktutil or net ads... but not both.



You can use both, but I try to avoid confusion (not sure if I am successful 
with it)



In fact, after installation of squid I did the way via msktutil but ntlm
authentication didn't work afterwards.
Maybe it was because of the client cache I missunderstood.



No this is probably because msktutil changed the random account password and 
samba's secrets.tdb database on the Unix server doesn't know about the 
changed password - so are out of sync.




If you say my installation will run into miss-behaviours of my keys 
(mkstutil

and net ads at the same time) then I will try to
- delete kerberos key on windows client
- use either msktutil or net ads

Maybe I can share my experience again.



Thank you


Thanks a lot
Andrew



Markus

Re: [squid-users] transparent integration with proxy on router

2009-09-28 Thread Todd Nine 
Thanks for the help!  I read over the rules and it was quite easy to set 
up what I needed once I had the right directive.  I simply set up the 
following.


#Set up our ACL for high throughput sites
acl high_throughput dstdomain .amazonaws.com

#Bind high throughput to the wireless interface
tcp_outgoing_address 116.90.140.xx high_throughput

However we're having a side effect issue.  Our router box is a bit old 
(an old P4), and we can't keep up with the squid demands due to the 
number of users with 2 GB of ram.  Is there a directive that I can tell 
squid not to proxy connections unless they meet the "high_throughput" 
acl?  I looked and couldn't find any bypass directives that met what I 
needed.


Thanks,
Todd

Amos Jeffries wrote:

On Mon, 28 Sep 2009 16:21:16 +1300, Todd Nine 
wrote:
  

Hi all,
  I'm using squid on a pfSense router we've built.  We have 2 
connections, one we pay for usage (DSL) and one we do not (Wireless).  
We use Amazon S3 extensively at work.  We've been attempting to route 
all traffic over the wireless via an IP range, but as S3 can change IPs, 
this doesn't work and we end up with a large bill for our DSL.  Is it 
possible to have squid route connections via a specific interface if a 
hostname such as "amazonaws.com" is in the HTTP request header?


Thanks,
Todd



Yes you can.

Find an IP assigned to the interface you want traffic to go out. Use the
tcp_outgoing_addr directive and ACLs that match the requests to make sure
all the requests to that domain are assigned that outgoing address.  Then
make sure the OS sends traffic from that IP out the right interface.

Amos
  


--
todd
SENIOR SOFTWARE ENGINEER

todd nine | spidertracks ltd |  117a the square
po box 5203 | palmerston north 4441 | new zealand
P: +64 6 353 3395 | M: +64 210 255 8576 
E: t...@spidertracks.co.nz  
W: www.spidertracks.com

Re: [squid-users] SSL Reverse Proxy testing With Invalid Certificate, can it be done.

2009-09-28 Thread Chris Robertson 

Dean Weimer wrote:

I am trying to setup a test with an SSL reverse proxy on an intranet site, I 
currently have a fake self signed certificate and the server is answering on 
the HTTP side just fine, and answering on the HTTPS however I get a (92) 
protocol error returned from the proxy when trying to access it through HTTPS.

I have added the following lines for the HTTPS option

https_port 443 accel cert=/usr/local/squid/etc/certs/server.crt 
key=/usr/local/squid/etc/certs/server.key defaultsite=mysite vhost

cache_peer 10.20.10.76 parent 443 0 no-query originserver ssl 
sslflags=DONT_VERIFY_PEER,DONT_VERIFY_DOMAIN name=secure_mysite

From the log I can see the error is caused by the invalid certificate.

2009/09/25 11:38:07| SSL unknown certificate error 18 in...
2009/09/25 11:38:07| fwdNegotiateSSL: Error negotiating SSL connection on FD 
15: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify 
failed (1/-1/0)

Is there a way that I can tell it to go ahead and trust this fake certificate 
during testing while I wait for the actual certificate that is valid, to be 
issued.
  


Perhaps http://www.squid-cache.org/Doc/config/sslproxy_flags/



Thanks,
 Dean Weimer
 Network Administrator
 Orscheln Management Co
  


Chris

Re: [squid-users] hardware specification

2009-09-28 Thread Chris Robertson 

HRINS wrote:

hi,
Im new with squid cache, my hardware specification server is 500 GByte hard
disk,8G ram. 
I have 1000 user about 400 active user so is it better to upgrade my

hardware or this is enough ?? I forget to mention the rate bandwidth is
12Mbps (from VSAT) 
  


What you have is likely fine.

Some suggestions:
* Don't try to fill the whole 500GB drive from the start.  Start small 
(100 GB) and grow if memory utilization allows.
* With satellite connections keep an eye on your Reserved/Free File 
Descriptor count.  Using a COSS dir for small files can help greatly, 
but is only supported on 2.7.

* Adjust your maximum_object_size to cache 1GB objects (or larger).

Chris

Re: [squid-users] Re: Re: Re: Re: squid_kerb_auth.... Key Version number?

2009-09-28 Thread Mrvka Andreas 
Hello Markus,

I thought there will be more changes in wiki than what you have written.

You write about either use msktutil or net ads... but not both.

In fact, after installation of squid I did the way via msktutil but ntlm 
authentication didn't work afterwards.
Maybe it was because of the client cache I missunderstood.


If you say my installation will run into miss-behaviours of my keys (mkstutil 
and net ads at the same time) then I will try to 
- delete kerberos key on windows client
- use either msktutil or net ads

Maybe I can share my experience again.

Thanks a lot
Andrew



Am Sonntag, 27. September 2009 22:30:18 schrieb Markus Moeller:
> Andrew,
> 
>I added more details to the wiki for cases where Samba is used too.  I
> hope this helps.
> 
> Regards
> Markus
> 
> "Mrvka Andreas"  wrote in message
> news:200909250845.48301@tuv.at...
> 
> > Agreed.
> >
> > So if I read your mail correctly you want to say:
> > - net ads join uses _computer-name_ to identify the authentication scheme
> > - msktutil (kerberos) only watches at the _service_ (http,cifs,...)
> >
> > The HowTo should look like:
> > 1.
> > use net ads join to talk via computer-name with AD
> >
> > 2
> > use msktutil _with a non-existend computer-name_ so that the associated
> > HOST/ can not correlate with net ads join
> > Only the sericePrincipal HTTP/ is important for squid/kerberos.
> >
> >
> > Have I understood you in the right way?
> > And will it work to use a non-existend hostname, or will msktutil fail?
> >
> > :-)
> >
> > The best way would be - the client sends an NTLM token and
> > squid_kerb_auth does the rest. :-)
> >
> >
> > Thanks for support.
> > I can imagine lots of other squid-users use net ads join and want to
> > implement
> > kerberos too.
> >
> > Regards
> > Andrew
> >
> > Am Freitag, 25. September 2009 01:07:44 schrieb Markus Moeller:
> >> "Henrik Nordstrom"  wrote in message
> >> news:1253822657.5592.1.ca...@localhost.localdomain...
> >>
> >> > tor 2009-09-24 klockan 10:09 +0200 skrev Mrvka Andreas:
> >> >> You are right - I have to use NTLM too because there are many IE 6
> >> >> around.
> >> >> But I use the same name for kerberos_auth and ntlm_auth
> >> >> (kerberos - samba/winbind)
> >> >> How should I configure a browser setting then? I want to set only one
> >> >> proxy
> >> >> server.
> >> >
> >> > Hmm.. I then suspect the HTTP ticket will get mismatch again in some
> >> > time when the computer account is renewed by Samba.
> >>
> >> I think so too.  Let me try to explain. Each entry in AD has a key
> >> associated with it. For a user account the key is based on the user
> >>  password and for a computer it is based on a random password. As you
> >> may have seen each entry in AD has also a serviceprincipalname
> >> attribute. This
> >>  attribute is used to associate a Kerberos principal with a key. You
> >> will see a computer account has usually a HOST/ host/fqdn
> >> serviceprincipal name and HTTP/fqdn if IIS is installed and cifs/fqdn
> >> for fileshares.
> >>
> >> net ads join creates an entry in AD with a random password with
> >>  CN=hostname. If you use msktutil with --computer-name hostname the same
> >> AD
> >>  entry will be used and since both commands will  set a random password
> >> you
> >>  will get conflicts. For Kerberos the computer name doesn't matter (only
> >>  the serviceprinciplname attribute is important) why you should use
> >>  msktutil with any computer name (e.g. -http) to avoid
> >> the conflict.
> >>
> >> Additionally msktutil sets the userprincipalname when you use --upn. The
> >> userprincipalname is used to authenticate a principal (user or other
> >> e.g. HTTP/) via kinit.  So if you use msktutil as described kinit
> >> -kt  HTTP/ will authenticate HTTP/ with the key (=
> >> encrypted random password) stored in the keytab.
> >>
> >> > If that's the case then I also guess you should be able to
> >> > automatically
> >> > renew the HTTP ticket using the Samba keytab however. But Kerberos is
> >> > not my main field of expertise..
> >> >
> >> > Regards
> >> > Henrik
> >>
> >> Regards
> >> Markus
>

Re: [squid-users] not caching enough

2009-09-28 Thread ant2ne 

Ok, it has come to my attention that /cache only grwos when I run a report
using sarg. So it maybe that my proxy server is working, but only as a proxy
and not as a web cache proxy. Below is my squid.conf file with the comment #
invert grepped out. Please review and tell me what it is that I need to
change to turn this proxy server into a web cache server.

below is me squid.conf
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl purge method PURGE
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
acl our_networks src 10.60.140.0/24
http_access allow our_networks
http_access allow localhost
http_access allow all
http_access deny all
icp_access allow all
http_port 3128
hierarchy_stoplist cgi-bin ?
cache_dir ufs /cache 50 256 256
maximum_object_size 32768 KB
access_log /var/log/squid/access.log squid
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
refresh_pattern ^ftp:   144020% 10080
refresh_pattern ^gopher:14400%  1440
refresh_pattern .   0   20% 4320
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
extension_methods REPORT MERGE MKACTIVITY CHECKOUT
hosts_file /etc/hosts
coredump_dir /var/spool/squid
visible_hostname AHSPX01
-- 
View this message in context: 
http://www.nabble.com/not-caching-enough-tp25530445p25645183.html
Sent from the Squid - Users mailing list archive at Nabble.com.

Re: [squid-users] not caching enough

2009-09-28 Thread Matus UHLAR - fantomas 
On 24.09.09 10:45, ant2ne wrote:
> This web cache proxy server is a dedicated machine running ubuntu 64bit OS
> (no gui) with 5gigs of RAM and a 1TB drive dedicated to the cache. Only
> using 500GB currently. (The OS is on a different 80GB drive) Accordign to
> the math of 10MB of RAM per 1GB of Disk space that would put me right at 5GB
> of RAM.

well, the 5GB of tram will be used only for cache indexes. You may need
additional memory for OS, cache etc. or your system may start swapping and
get unusable.

> Yesterday I increased the maximum_object_size value to 32MB from the default
> of 4MB and haven't noticed any difference in the cache size. I think I'll
> push it up around 50MB.

How much traffic goes through your cache?

Note that

- there's much of uncacheable content on the web
  (much of it could be cached if the developers cared)

- squid only tried to cache fetched content.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Nothing is fool-proof to a talented fool.

Re: [squid-users] POST NONE://

2009-09-28 Thread Mario Remy Almeida 
Hi Amos,

Thanks for that, My problem is solved.

Is there any way to by-pass such problems. I mean for known source IP if
HTTP headers are not set then still it is pass through.

//Remy

Amos Jeffries wrote:
> Mario Remy Almeida wrote:
>> Hi Amos,
>>
>> Thanks for your reply.
>>
>> You mean the length is less then what is required?
>>
>
> No there is an HTTP header "Content-Length:" which is missing from the
> POST request.
>
>> When you pass the below SOAP message I get the error, But when it is
>> passed directly to the jboss applications server the request is served
>> correctly.
>>
>> What is wrong the Header settings in the soap message or do i need to do
>> some config changes in squid.conf file
>
> The SOAP message below looks like data inside the body. The missing
> bit is in the wrapping HTTP headers. They are generated by the client
> software.
>
> Amos
>
>>
>> === SOAP MESSAGE ==
>> > xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/";
>> xmlns:env="http://schemas.xmlsoap.org/soap/envelop/";
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>> xmlns:enc="http://schemas.xmlsoap.org/soap/encoding/";
>> xmlns:xsd="http://www.w3.org/2001/XMLSchema";>
>> > xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";>WSUSERpass43>
>> xmlns:ns2="http://www.opentravel.org/OTA/2003/05";
>> EchoToken="WWW0909271406222" PrimaryLangID="en-us" SequenceNmbr="1"
>> TimeStamp="2009-09-27T02:06:22"
>> TransactionIdentifier="">> Type="9">> Type="9">>
>> ID="18496815"
>> Type="14">2009-10-30T00:00:00>
>> xmlns:ns1="http://www.isaaviation.com/thinair/webservices/OTA/Extensions/2003/05";>truetruetruetrue
>>
>>
>>
>>
>> //Remy
>>
>> Amos Jeffries wrote:
>>> Mario Remy Almeida wrote:
 Hi All,

 would like to know what is the reason that i get NONE:// in the
 access.log file as below

 1254046127.530  0 195.229.115.202 TCP_DENIED/411 1757 POST
 NONE:// -
 NONE/- text/html

 my squid proxy acts like a reverse proxy.

 A valid request is sent from the above IP
>>> 411 status code is failure to pass a basic validity test.
>>> This one was a test for Content-Length: header on POST requests.
>>>

 Could some one help be in solving the problem.


 My setup

 Request from Internet -> Squid Reverse proxy(A) -> Squid reverse
 proxy(B) -> Jboss Applications Server.

>>> The NONE:// means no the request did not complete, it did not even get
>>> far enough to determine if it was a HIT or MISS on the URL.
>>> This is due to the required header making Squid abort its processing
>>> immediately.
>>>
>>> Amos
>>
>
> Amos

--
Disclaimer and Confidentiality


This material has been checked for  computer viruses and although none has
been found, we cannot guarantee  that it is completely free from such problems
and do not accept any  liability for loss or damage which may be caused.
Please therefore  check any attachments for viruses before using them on your
own  equipment. If you do find a computer virus please inform us immediately
so that we may take appropriate action. This communication is intended  solely
for the addressee and is confidential. If you are not the intended recipient,
any disclosure, copying, distribution or any action  taken or omitted to be
taken in reliance on it, is prohibited and may be  unlawful. The views
expressed in this message are those of the  individual sender, and may not
necessarily be that of ISA.