Re: [squid-users] Squid and Perl and DENIED

2007-10-14 Thread Chris Robertson

Joe wrote:

Chris--

Thanks so much for helping.  My knowledge of squid is
limited to what it took to get it up and running, so
experienced help is invaluable.  (btw, The two configs
were the same, but I cut those deny lines out just out
of curiousity b/c it looks to me like Squid-b is
denying my requests without even passing them on, so I
tried loosening the acl rules.)
  


With the new information given below, I think the problem may not be 
caused by Squid.



Here's what squid-A is allowing through (as
requested):

ticketmaster test:

1192289991.372831 24.1.8.252 TCP_MISS/200 40211
GET https://www.ticketmaster.ca/checkout/order? -
DIRECT/209.104.58.144 text/html
  


Fascinating.  It's an actual fully recognizable GET.  Usually, (when a 
browser hits a secure site) you only get the host name in the logs 
(example:
1192312024.368   2958 209.165.134.47 TCP_MISS/200 28087 CONNECT 
login.yahoo.com:443 - DIRECT/209.73.168.74 -)



1192289983.189   1549 24.1.8.252 TCP_MISS/200 9970
POST
https://www.ticketmaster.ca/checkout/reserve/fMZQyGUS6wKOSfQ5vMbYq_1NrvDR_mW4ySFWjti-a09EeTMHBt01oJHgkTaYmSKO1DEVuSa6CmlF_Gq4PlCIig
- DIRECT/209.104.58.144 text/html


wellsfargo.com test

1192297304.994   1162 67.163.91.153 TCP_MISS/200 17212
GET https://www.wellsfargo.com/ -
DIRECT/151.151.88.133 text/html

(Squid-A is on CentOS 4.2, Squid-b RH release 9)

  


Hmm... A difference.  RH 9 is getting up there in years (it's only four 
years old, but quite a bit changed in that time).  I wonder if the 
different network stacks (heck, RH9 uses a 2.4 kernel!) would have 
something to do with it.



Any thoughts?  My knowledge of SSL protocols is also
enough to get by.  Why do you think squid-A
shouldn't be okay with GET/POST over https?
  


It's not so much that squid would be okay with a GET over HTTPs, more 
that the website being accessed wouldn't care for a GET over a secure 
channel.  Try the following:


telnet www.ticketmaster.ca 80
GET / HTTP/1.0
[hit enter again]

You should see a bunch of HTML source.

Now try:

telnet www.ticketmaster.ca 443
GET / HTTP/1.0
[hit enter again]

The response is a 400 error explaining the problem of using plain HTTP 
on an SSL-enabled server port.



Thanks!

-Joe
  


You might try using ssldump (www.rtfm.com/ssldump/) to watch the 
connection from each Squid to the respective websites.  Beyond that, I 
can't really say.  I truly don't understand how/why the connections 
through Squid-A work!


Chris


Re: [squid-users] DISKD Autoremoved

2007-10-14 Thread Guido Serassio

Hi,

At 02.53 14/10/2007, Juan C. Crespo R. wrote:

hi

   Does anyone knows why when I try to install squid with diskd, 
this option autoremove itself?


./configure --prefix=/usr/local/squid --enable-async-io=128 
--enable-storio=diskd,ufs --enable-


There is a typo: it should be  --enable-storeio.

Regards

Guido



-

Guido Serassio
Acme Consulting S.r.l. - Microsoft Certified Partner
Via Lucia Savarino, 1   10098 - Rivoli (TO) - ITALY
Tel. : +39.011.9530135  Fax. : +39.011.9781115
Email: [EMAIL PROTECTED]
WWW: http://www.acmeconsulting.it/



Re: [squid-users] ACL help: blocking non-html objects from particular domains

2007-10-14 Thread Amos Jeffries

Craig Skinner wrote:

I'm attempting to use ACLs to block non-HTML objects from particular
domains. i.e: users should be able to see the html, but not the images.

Tried various forms of and always end up will all or nothing:

acl suspect-domains dstdom_regex /etc/squid/suspect-domains.acl


please, PLEASE use dstdomain instead of regex.


acl ok-mime-types rep_mime_type -i ^text/html$
acl ok-mime-types rep_mime_type -i text/html


First match falls inside the second. One of them is redundant.



# 1
#http_access allow ok-mime-types
#http_access allow suspect-domains


allow mime types + allow suspect domains
--- you wanted blocking. no blocking there.



# 2
#http_access allow suspect-domains ok-mime-types


better, and should work if you have no other allows' involving 
suspect-domains.




# 3
#http_access deny suspect-domains !ok-mime-types
#http_access allow suspect-domains


This (3) should be working. It's exactly what you specified as your 
requirements above. Maybe to be extra paranoid allow suspect-domains 
ok-mime-types', but not strictly needed.




http_access deny all

What am I missing here?


Nothing that I can see from those.
It should work. What does cache.log / access.log say when (3) is used?

Amos


Re: [squid-users] 2.6-16 compile error on freebsd

2007-10-14 Thread Michel Santos


 I get a compile error with squid-2.6-STABLE-16 as follows


...

 ./cf_gen cf.data ./cf.data.depend
 *** Signal 10

 Stop in /usr/local/squid/squid-2.6.STABLE16/src.
 *** Error code 1




is it possibly a compiler problem?

gcc 4.2.1 is the only difference on FreeBSD7 I can find ( on the machines
FreeBSD6 with gcc 3.4.6 it compiles fine)

on the other hand, squid compiled with gcc 3.4.6 on FreeBSD6 runs fine on
FreeBSD7



Michel
...





Tecnologia Internet Matik http://info.matik.com.br
Sistemas Wireless para o Provedor Banda Larga
Hospedagem e Email personalizado - e claro, no Brasil.




Re: [squid-users] 2.6-16 compile error on freebsd

2007-10-14 Thread Thomas-Martin Seck
* Michel Santos ([EMAIL PROTECTED]):

 
 
  I get a compile error with squid-2.6-STABLE-16 as follows
 
 
 ...
 
  ./cf_gen cf.data ./cf.data.depend
  *** Signal 10
 
  Stop in /usr/local/squid/squid-2.6.STABLE16/src.
  *** Error code 1
 
 
 
 
 is it possibly a compiler problem?

This is a bug in cf_gen that only manifests itself on FreeBSD 7
(either because the new malloc implementation handles things
differently in general or because its internal debugging code was
active until FreeBSD-7 was officially branched in CVS). Please look
at http://www.squid-cache.org/Versions/v2/2.6/changesets/ for the
patch to fix this. [Shameless plug: or just use the port, it contains
the fix.]


Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Amos Jeffries

Haytham KHOUJA (devnull) wrote:

Hello,
The purpose of this thread is to join forces to have the best Squid 
configuration for generic affordable Intel machines available by major 
vendors (Dell/HP...) specifically for ISPs and corporations that want a 
basic setup but with optimal response and throughput and maximizing 
bandwidth savings.
I work for an important ISP and I currently replaced 2 NetApp NetCache 
with 3 Dell 2950 hooked up on a Foundry Switch for Load Balancing.
I used tproxy to enable IP Spoofing to IP spoofing the outgoing address 
with some configurations on the Cisco core router, I had to compile 
iptables and tproxy on a Debian kernel source (2.6.18)


I've read almost every single thread on Optimizing Squid and Linux and 
want to share my setup with you.
I do have some questions, clarifications and bugs but overall the 
performance is pretty impressive. (Yes, much better than the NetApps)


What i want to do is since i have 8 GB of RAMs, i want to store more hot 
objects in the RAMs to maximize Memory hit ratio, but with my setup, 
Squid doesn't
go above 2GB~3GB of usage. (Remember, that there are no other heavy 
processes on the machine).


You will need a 64-bit enabled squid to go higher than 2GB.



If i knew beforehand that Squid doesn't make use of SMP, i wouldn't have 
bought Dual Quad Core and would have invested in Intel CPUs with 8mb of 
Cache, but what's done is done :)


Before i had Squid go down because of File Delimiters and maximum open 
files and ip_conntrac fill up, i fixed both with some iptables and 
sysctl configuration.
Now i'm hitting a Oct 14 01:17:06 proxy4 squid[8883]: assertion failed: 
diskd/store_io_diskd.c:384: !diskdstate-flags.close_request Error, so 
Squid kills and restarts (which flushes the Memory cache).


I'm looking forward for some contributions, idea sharing, knowledge 
correcting to make this setup a standard setup for large scale, well 
optimized and high performant Squid for future tweakings. I hope this 
configuration would be then uploaded to the Squid wiki.


Post your squid.conf to
  http://squid.treenet.co.nz/cf.check/
and review the results. I've pointed out the biggest worries below.



Here's my setup:
Dell 2950
Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives

I have 3 cache_dir on separate drives and I formated the 3 disks with 
ReiserFS:

   /dev/sdb1   /CACHE1 reiserfs notail,noatime 0 0
   /dev/sdc1   /CACHE2 reiserfs notail,noatime 0 0
   /dev/sdd1   /CACHE3 reiserfs notail,noatime 0 0

I run Debian GNU/Linux Etch and compiled Squid with the following:
Squid Cache: Version 2.6.STABLE16
configure options:  '--bindir=/usr/bin' '--sbindir=/usr/sbin/' 
'--sysconfdir=/etc' '--enable-icmp' '--enable-snmp' '--enable-async-io' 
'--enable-linux-netfilter' '--enable-linux-tproxy' '--with-dl' 
'--with-large-files' '--enable-large-cache-files' '--with-maxfd=100' 
'--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll' 
'--disable-ident-lookups' '--enable-removal-policies=heap' 
'CFLAGS=-DNUMTHREADS=120'


As you can see i have the following modules enabled: linux-tproxy, 
diskd, epoll, and removal policies.
/dev/epoll improves network I/O performance, Diskd separates disk I/O to 
separate processes (which reduces process locking from Squid to write on 
disks), and read benchmarks for memory and disk removal policies.


aufs does a better job, particularly where threads are available and is 
not quite so broken as diskd.



My /etc/squid.conf is composed of the following:

http_port 80 transparent tproxy
tcp_outgoing_address IP of the Machine
:: Those are for IP Spooding and Transparency

via off
forwarded_for off
:: Those are for total transparency, remote hosts will never guess that 
the request came from a proxy


IIRC, theres more than this needed for complete silence. They just 
replace the Via and Forwarded-For with text 'unknown'. still leaving the 
headers in place for anon-proxy identification.




cache_mem 600 MB
:: A bit confused about this, When i go higher than 2GB, Squid kills 
with a out of memory error. I have 8GB and want to maximize the use of 
it.


cache_effective_user nobody
cache_effective_group nogroup
:: Security and bla bla


This is the default UID. If this is going to be a standard config these 
MUST not be explicitly set.
Also when GID is configured as above, will in fact cause a 
squid-specific deviation from the configured OS-level security policy.


They are no longer to be used, unless the machine-specific setup 
requires it AND the admin knows how to setup for them properly.



cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF
:: Very objective, you can google about them

cache_dir diskd /CACHE1 61440 16 256 Q1=144 Q2=128
cache_dir diskd /CACHE2 61440 16 256 Q1=144 Q2=128
cache_dir diskd /CACHE3 61440 16 256 Q1=144 Q2=128
:: DISKD configuration, i'm only using 60GB of each disk

cache_access_log /var/log/squid/access.log



Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Haytham KHOUJA (devnull)

Dear Amos,
Thank you for your reply, check my comments:

Amos Jeffries wrote:

Haytham KHOUJA (devnull) wrote:

Hello,
The purpose of this thread is to join forces to have the best Squid 
configuration for generic affordable Intel machines available by 
major vendors (Dell/HP...) specifically for ISPs and corporations 
that want a basic setup but with optimal response and throughput and 
maximizing bandwidth savings.
I work for an important ISP and I currently replaced 2 NetApp 
NetCache with 3 Dell 2950 hooked up on a Foundry Switch for Load 
Balancing.
I used tproxy to enable IP Spoofing to IP spoofing the outgoing 
address with some configurations on the Cisco core router, I had to 
compile iptables and tproxy on a Debian kernel source (2.6.18)


I've read almost every single thread on Optimizing Squid and Linux 
and want to share my setup with you.
I do have some questions, clarifications and bugs but overall the 
performance is pretty impressive. (Yes, much better than the NetApps)


What i want to do is since i have 8 GB of RAMs, i want to store more 
hot objects in the RAMs to maximize Memory hit ratio, but with my 
setup, Squid doesn't
go above 2GB~3GB of usage. (Remember, that there are no other heavy 
processes on the machine).


You will need a 64-bit enabled squid to go higher than 2GB.


Yea, I hope i'll be able to replace the CPUs


If i knew beforehand that Squid doesn't make use of SMP, i wouldn't 
have bought Dual Quad Core and would have invested in Intel CPUs with 
8mb of Cache, but what's done is done :)


Before i had Squid go down because of File Delimiters and maximum 
open files and ip_conntrac fill up, i fixed both with some iptables 
and sysctl configuration.
Now i'm hitting a Oct 14 01:17:06 proxy4 squid[8883]: assertion 
failed: diskd/store_io_diskd.c:384: 
!diskdstate-flags.close_request Error, so Squid kills and restarts 
(which flushes the Memory cache).


I'm looking forward for some contributions, idea sharing, knowledge 
correcting to make this setup a standard setup for large scale, well 
optimized and high performant Squid for future tweakings. I hope this 
configuration would be then uploaded to the Squid wiki.


Post your squid.conf to
  http://squid.treenet.co.nz/cf.check/
and review the results. I've pointed out the biggest worries below.



Here's my setup:
Dell 2950
Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives

I have 3 cache_dir on separate drives and I formated the 3 disks with 
ReiserFS:

   /dev/sdb1   /CACHE1 reiserfs notail,noatime 0 0
   /dev/sdc1   /CACHE2 reiserfs notail,noatime 0 0
   /dev/sdd1   /CACHE3 reiserfs notail,noatime 0 0

I run Debian GNU/Linux Etch and compiled Squid with the following:
Squid Cache: Version 2.6.STABLE16
configure options:  '--bindir=/usr/bin' '--sbindir=/usr/sbin/' 
'--sysconfdir=/etc' '--enable-icmp' '--enable-snmp' 
'--enable-async-io' '--enable-linux-netfilter' 
'--enable-linux-tproxy' '--with-dl' '--with-large-files' 
'--enable-large-cache-files' '--with-maxfd=100' 
'--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll' 
'--disable-ident-lookups' '--enable-removal-policies=heap' 
'CFLAGS=-DNUMTHREADS=120'


As you can see i have the following modules enabled: linux-tproxy, 
diskd, epoll, and removal policies.
/dev/epoll improves network I/O performance, Diskd separates disk I/O 
to separate processes (which reduces process locking from Squid to 
write on disks), and read benchmarks for memory and disk removal 
policies.


aufs does a better job, particularly where threads are available and 
is not quite so broken as diskd.

I will recompile, use aufs and do more testing

My /etc/squid.conf is composed of the following:

http_port 80 transparent tproxy
tcp_outgoing_address IP of the Machine
:: Those are for IP Spooding and Transparency

via off
forwarded_for off
:: Those are for total transparency, remote hosts will never guess 
that the request came from a proxy


IIRC, theres more than this needed for complete silence. They just 
replace the Via and Forwarded-For with text 'unknown'. still leaving 
the headers in place for anon-proxy identification.



True, but this is used with tproxy for ip spoofing


cache_mem 600 MB
:: A bit confused about this, When i go higher than 2GB, Squid kills 
with a out of memory error. I have 8GB and want to maximize the use 
of it.


cache_effective_user nobody
cache_effective_group nogroup
:: Security and bla bla


So i can leave it to 2GB maximum? The rest of the OS will have the rest 
of the RAM for OS purposes.
This is the default UID. If this is going to be a standard config 
these MUST not be explicitly set.
Also when GID is configured as above, will in fact cause a 
squid-specific deviation from the configured OS-level security policy.


They are no longer to be used, unless the machine-specific setup 
requires it AND the admin knows how to setup for them properly.



cache_replacement_policy heap LFUDA

Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Tony Dodd

Hi,



You will need a 64-bit enabled squid to go higher than 2GB.


Yea, I hope i'll be able to replace the CPUs


How old are the 2950s?  AFAIK, those produced in the last 3-4 years have 
all been 64bit capable; you should only need to reinstall with a 64bit 
distro. =]


--
Tony Dodd, Systems Administrator

Last.fm | http://www.last.fm
Karen House 1-11 Baches Street
London N1 6DL

check out my music taste at:
http://www.last.fm/user/hawkeviper


Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Marcus Kool

Hi Haytham,

you stated that the current Squid server is faster than the Netcache boxes.
Just to do a fair and clear comparison (I and possibly others like to
have a more clear picture of how Netcache and Squid compare to each other):
can you give the Netcache spec. (model, version, memory size etc) and do you 
have
numbers like reqs/sec or delays ?

thanks
Marcus

Haytham KHOUJA (devnull) wrote:

Dear Amos,
Thank you for your reply, check my comments:

Amos Jeffries wrote:

Haytham KHOUJA (devnull) wrote:

Hello,
The purpose of this thread is to join forces to have the best Squid 
configuration for generic affordable Intel machines available by 
major vendors (Dell/HP...) specifically for ISPs and corporations 
that want a basic setup but with optimal response and throughput and 
maximizing bandwidth savings.
I work for an important ISP and I currently replaced 2 NetApp 
NetCache with 3 Dell 2950 hooked up on a Foundry Switch for Load 
Balancing.
I used tproxy to enable IP Spoofing to IP spoofing the outgoing 
address with some configurations on the Cisco core router, I had to 
compile iptables and tproxy on a Debian kernel source (2.6.18)


I've read almost every single thread on Optimizing Squid and Linux 
and want to share my setup with you.
I do have some questions, clarifications and bugs but overall the 
performance is pretty impressive. (Yes, much better than the NetApps)


What i want to do is since i have 8 GB of RAMs, i want to store more 
hot objects in the RAMs to maximize Memory hit ratio, but with my 
setup, Squid doesn't
go above 2GB~3GB of usage. (Remember, that there are no other heavy 
processes on the machine).


You will need a 64-bit enabled squid to go higher than 2GB.


Yea, I hope i'll be able to replace the CPUs


If i knew beforehand that Squid doesn't make use of SMP, i wouldn't 
have bought Dual Quad Core and would have invested in Intel CPUs with 
8mb of Cache, but what's done is done :)


Before i had Squid go down because of File Delimiters and maximum 
open files and ip_conntrac fill up, i fixed both with some iptables 
and sysctl configuration.
Now i'm hitting a Oct 14 01:17:06 proxy4 squid[8883]: assertion 
failed: diskd/store_io_diskd.c:384: 
!diskdstate-flags.close_request Error, so Squid kills and restarts 
(which flushes the Memory cache).


I'm looking forward for some contributions, idea sharing, knowledge 
correcting to make this setup a standard setup for large scale, well 
optimized and high performant Squid for future tweakings. I hope this 
configuration would be then uploaded to the Squid wiki.


Post your squid.conf to
  http://squid.treenet.co.nz/cf.check/
and review the results. I've pointed out the biggest worries below.



Here's my setup:
Dell 2950
Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives

I have 3 cache_dir on separate drives and I formated the 3 disks with 
ReiserFS:

   /dev/sdb1   /CACHE1 reiserfs notail,noatime 0 0
   /dev/sdc1   /CACHE2 reiserfs notail,noatime 0 0
   /dev/sdd1   /CACHE3 reiserfs notail,noatime 0 0

I run Debian GNU/Linux Etch and compiled Squid with the following:
Squid Cache: Version 2.6.STABLE16
configure options:  '--bindir=/usr/bin' '--sbindir=/usr/sbin/' 
'--sysconfdir=/etc' '--enable-icmp' '--enable-snmp' 
'--enable-async-io' '--enable-linux-netfilter' 
'--enable-linux-tproxy' '--with-dl' '--with-large-files' 
'--enable-large-cache-files' '--with-maxfd=100' 
'--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll' 
'--disable-ident-lookups' '--enable-removal-policies=heap' 
'CFLAGS=-DNUMTHREADS=120'


As you can see i have the following modules enabled: linux-tproxy, 
diskd, epoll, and removal policies.

/dev/epoll improves network I/O performance, Diskd separates disk I/O CP
to separate processes (which reduces process locking from Squid to 
write on disks), and read benchmarks for memory and disk removal 
policies.


aufs does a better job, particularly where threads are available and 
is not quite so broken as diskd.

I will recompile, use aufs and do more testing

My /etc/squid.conf is composed of the following:

http_port 80 transparent tproxy
tcp_outgoing_address IP of the Machine
:: Those are for IP Spooding and Transparency

via off
forwarded_for off
:: Those are for total transparency, remote hosts will never guess 
that the request came from a proxy


IIRC, theres more than this needed for complete silence. They just 
replace the Via and Forwarded-For with text 'unknown'. still leaving 
the headers in place for anon-proxy identification.



True, but this is used with tproxy for ip spoofing


cache_mem 600 MB
:: A bit confused about this, When i go higher than 2GB, Squid kills 
with a out of memory error. I have 8GB and want to maximize the use 
of it.


cache_effective_user nobody
cache_effective_group nogroup
:: Security and bla bla


So i can leave it to 2GB maximum? The rest of the OS will have the rest 
of the RAM for OS purposes.
This is the 

Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Haytham KHOUJA (devnull)

Hello Marcus,
I'll send all this info within this week as i'll perform many tests on a 
live large scale system, i'll submit them soon.

Please specify types of tests you'd want me to run

Marcus Kool wrote:

Hi Haytham,

you stated that the current Squid server is faster than the Netcache 
boxes.

Just to do a fair and clear comparison (I and possibly others like to
have a more clear picture of how Netcache and Squid compare to each 
other):
can you give the Netcache spec. (model, version, memory size etc) and 
do you have

numbers like reqs/sec or delays ?

thanks
Marcus

Haytham KHOUJA (devnull) wrote:

Dear Amos,
Thank you for your reply, check my comments:

Amos Jeffries wrote:

Haytham KHOUJA (devnull) wrote:

Hello,
The purpose of this thread is to join forces to have the best Squid 
configuration for generic affordable Intel machines available by 
major vendors (Dell/HP...) specifically for ISPs and corporations 
that want a basic setup but with optimal response and throughput 
and maximizing bandwidth savings.
I work for an important ISP and I currently replaced 2 NetApp 
NetCache with 3 Dell 2950 hooked up on a Foundry Switch for Load 
Balancing.
I used tproxy to enable IP Spoofing to IP spoofing the outgoing 
address with some configurations on the Cisco core router, I had to 
compile iptables and tproxy on a Debian kernel source (2.6.18)


I've read almost every single thread on Optimizing Squid and Linux 
and want to share my setup with you.
I do have some questions, clarifications and bugs but overall the 
performance is pretty impressive. (Yes, much better than the NetApps)


What i want to do is since i have 8 GB of RAMs, i want to store 
more hot objects in the RAMs to maximize Memory hit ratio, but with 
my setup, Squid doesn't
go above 2GB~3GB of usage. (Remember, that there are no other heavy 
processes on the machine).


You will need a 64-bit enabled squid to go higher than 2GB.


Yea, I hope i'll be able to replace the CPUs


If i knew beforehand that Squid doesn't make use of SMP, i wouldn't 
have bought Dual Quad Core and would have invested in Intel CPUs 
with 8mb of Cache, but what's done is done :)


Before i had Squid go down because of File Delimiters and maximum 
open files and ip_conntrac fill up, i fixed both with some iptables 
and sysctl configuration.
Now i'm hitting a Oct 14 01:17:06 proxy4 squid[8883]: assertion 
failed: diskd/store_io_diskd.c:384: 
!diskdstate-flags.close_request Error, so Squid kills and 
restarts (which flushes the Memory cache).


I'm looking forward for some contributions, idea sharing, knowledge 
correcting to make this setup a standard setup for large scale, 
well optimized and high performant Squid for future tweakings. I 
hope this configuration would be then uploaded to the Squid wiki.


Post your squid.conf to
  http://squid.treenet.co.nz/cf.check/
and review the results. I've pointed out the biggest worries below.



Here's my setup:
Dell 2950
Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives

I have 3 cache_dir on separate drives and I formated the 3 disks 
with ReiserFS:

   /dev/sdb1   /CACHE1 reiserfs notail,noatime 0 0
   /dev/sdc1   /CACHE2 reiserfs notail,noatime 0 0
   /dev/sdd1   /CACHE3 reiserfs notail,noatime 0 0

I run Debian GNU/Linux Etch and compiled Squid with the following:
Squid Cache: Version 2.6.STABLE16
configure options:  '--bindir=/usr/bin' '--sbindir=/usr/sbin/' 
'--sysconfdir=/etc' '--enable-icmp' '--enable-snmp' 
'--enable-async-io' '--enable-linux-netfilter' 
'--enable-linux-tproxy' '--with-dl' '--with-large-files' 
'--enable-large-cache-files' '--with-maxfd=100' 
'--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll' 
'--disable-ident-lookups' '--enable-removal-policies=heap' 
'CFLAGS=-DNUMTHREADS=120'


As you can see i have the following modules enabled: linux-tproxy, 
diskd, epoll, and removal policies.
/dev/epoll improves network I/O performance, Diskd separates disk 
I/O CP
to separate processes (which reduces process locking from Squid to 
write on disks), and read benchmarks for memory and disk removal 
policies.


aufs does a better job, particularly where threads are available and 
is not quite so broken as diskd.

I will recompile, use aufs and do more testing

My /etc/squid.conf is composed of the following:

http_port 80 transparent tproxy
tcp_outgoing_address IP of the Machine
:: Those are for IP Spooding and Transparency

via off
forwarded_for off
:: Those are for total transparency, remote hosts will never guess 
that the request came from a proxy


IIRC, theres more than this needed for complete silence. They just 
replace the Via and Forwarded-For with text 'unknown'. still leaving 
the headers in place for anon-proxy identification.



True, but this is used with tproxy for ip spoofing


cache_mem 600 MB
:: A bit confused about this, When i go higher than 2GB, Squid 
kills with a out of memory error. I have 8GB and want to maximize 
the 

Re: [squid-users] DISKD Autoremoved

2007-10-14 Thread Juan C. Crespo R.

Thank you


Guido Serassio escribió:

Hi,

At 02.53 14/10/2007, Juan C. Crespo R. wrote:

hi

   Does anyone knows why when I try to install squid with diskd, this 
option autoremove itself?


./configure --prefix=/usr/local/squid --enable-async-io=128 
--enable-storio=diskd,ufs --enable-


There is a typo: it should be  --enable-storeio.

Regards

Guido



-

Guido Serassio
Acme Consulting S.r.l. - Microsoft Certified Partner
Via Lucia Savarino, 1   10098 - Rivoli (TO) - ITALY
Tel. : +39.011.9530135  Fax. : +39.011.9781115
Email: [EMAIL PROTECTED]
WWW: http://www.acmeconsulting.it/





Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

2007-10-14 Thread Amos Jeffries
 Dear Amos,
 Thank you for your reply, check my comments:

 Amos Jeffries wrote:
 Haytham KHOUJA (devnull) wrote:

snip

 cache_mem 600 MB
 :: A bit confused about this, When i go higher than 2GB, Squid kills
 with a out of memory error. I have 8GB and want to maximize the use
 of it.

 cache_effective_user nobody
 cache_effective_group nogroup
 :: Security and bla bla

 So i can leave it to 2GB maximum? The rest of the OS will have the rest
 of the RAM for OS purposes.

I was talking below about the cache_effective_* above. I know very little
about the memory limits in squid, not having tested any yet.

 This is the default UID. If this is going to be a standard config
 these MUST not be explicitly set.
 Also when GID is configured as above, will in fact cause a
 squid-specific deviation from the configured OS-level security policy.

 They are no longer to be used, unless the machine-specific setup
 requires it AND the admin knows how to setup for them properly.

 cache_replacement_policy heap LFUDA
 memory_replacement_policy heap GDSF
 :: Very objective, you can google about them

 cache_dir diskd /CACHE1 61440 16 256 Q1=144 Q2=128
 cache_dir diskd /CACHE2 61440 16 256 Q1=144 Q2=128
 cache_dir diskd /CACHE3 61440 16 256 Q1=144 Q2=128
 :: DISKD configuration, i'm only using 60GB of each disk

 cache_access_log /var/log/squid/access.log

 Obsolete option. Use access_log with same parameters instead.

 Which is obsolete?

cache_access_log, it is deprecated from 2.6 and not guaranteed to be
around much longer.


Amos