[squid-users] Re: kid2| WARNING: disk-cache maximum object size is unlimited but mem-cache maximum object size is 32.00 KB

[Linda Walsh]

Noticed a few items I would try to simplify (these are suggestions
from my own experience...)...

1) for storio, don't use "rock" -- it limits items on disk to 32k and on 
a 486 you aren't

going to want what it was designed for (multi-core systems).



2) get rid of things you don't need i.e., are you really using delay pools?
cache-digests? icap?, esi?? Are you only trying to run it as
a "transparent proxy"  (enable-linux-netfilter)?

if you have nothing working... try tossing the auth and acl helpers on your
first test version...  Once you get something working, add your needed 
features

back in.

Given that it's your first & only squid proxy, I doubt you need cache 
digests in
your test version, for example (I believe it is used for sharing content 
between

squid servers... )...   simplify until you get something working...




On 10/25/2013 1:58 PM, Ahmad wrote:

hi  i compiled squid 3.3.8 with below :

Squid Cache: Version 3.3.8
configure options:  '--build=i486-linux-gnu' '--prefix=/usr'
'--includedir=/include' '--mandir=/share/man' '--infodir=/share/info'
'--sysconfdir=/etc' '--enable-cachemgr-hostname=drx' '--localstatedir=/var'
'--libexecdir=/lib/squid' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--disable-silent-rules' '--srcdir=.'
'--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
'--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8'
'--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap'
'--enable-delay-pools' '--enable-cache-digests' '--enable-underscores'
'--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
'--enable-ntlm-auth-helpers=smb_lm'
'--enable-digest-auth-helpers=ldap,password'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
'--enable-arp-acl' '--enable-esi' '--disable-translation'
'--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid'
'--with-filedescriptors=131072' '--with-large-files'
'--with-default-user=squid' '--enable-linux-netfilter'
'build_alias=i486-linux-gnu' 'CFLAGS=-g -O2 -g -Wall -O2' 'LDFLAGS='
'CPPFLAGS=' 'CXXFLAGS=-g -O2 -g 
-Wall -O2' --enable-ltdl-convenience



i follwed example exist in :
http://wiki.squid-cache.org/Features/SmpScale 


the problem is that there is warning occurs and then squid get down ,
here is it !!
*kid2| WARNING: disk-cache maximum object size is unlimited but mem-cache
maximum object size is 32.00 KB*

logs say that squid is down , but i can access squid by port , but as i
think there is no caching occuring in hardsiks !!! tr has fixed sizeand all
logs are tcp miss  not tcp hit !!!

i googled alot but no benefit

here is log file when i start squid :
[root@DataBase ~]# tailf /var/log/squid/backend.cache.log 
2013/10/25 22:47:39 kid2| Logfile: closing log

stdio:/var/log/squid/backend.access.log
2013/10/25 22:47:39 kid4| Open FD UNSTARTED 8 DNS Socket IPv6
2013/10/25 22:47:39 kid4| Open FD UNSTARTED 9 DNS Socket IPv4
2013/10/25 22:47:39 kid4| Open FD READING  13 
2013/10/25 22:47:39 kid2| Open FD UNSTARTED 8 DNS Socket IPv6

2013/10/25 22:47:39 kid2| Open FD UNSTARTED 9 DNS Socket IPv4
2013/10/25 22:47:39 kid2| Open FD READING  13 
2013/10/25 22:47:39 kid3| Squid Cache (Version 3.3.8): Exiting normally.

2013/10/25 22:47:39 kid2| Squid Cache (Version 3.3.8): Exiting normally.
2013/10/25 22:47:39 kid4| Squid Cache (Version 3.3.8): Exiting normally.



2013/10/25 22:50:24 kid4| Preparing for shutdown after 0 requests
2013/10/25 22:50:24 kid4| Waiting 30 seconds for active connections to
finish
2013/10/25 22:50:24 kid4| Shutdown: NTLM authentication.
2013/10/25 22:50:24 kid4| Shutdown: Negotiate authentication.
2013/10/25 22:50:24 kid4| Shutdown: Digest authentication.
2013/10/25 22:50:24 kid4| Shutdown: Basic authentication.
2013/10/25 22:50:24 kid2| Preparing for shutdown after 0 requests
2013/10/25 22:50:24 kid3| Preparing for shutdown after 0 requests
2013/10/25 22:50:24 kid2| Waiting 30 seconds for active connections to
finish
2013/10/25 22:50:24 kid3| Waiting 30 seconds for active connections to
finish
2013/10/25 22:50:24 kid2| Closing HTTP port 127.0.0.1:4002
2013/10/25 22:50:24 kid2| Waiting 30 seconds for active connections to
finish
2013/10/25 22:50:24 kid3| Waiting 30 seconds for active connections to
finish
2013/10/25 22:50:24 kid2| Closing HTTP port 127.0.0.1:4002
2013/10/25 22:50:24 kid3| Closing HTTP port 127.0.0.1:4003
2013/10/25 22:50:24 kid2| Shutdown: NTLM authentication.
2013/10/25 22:50:24 kid3| Closing HTTP port 127.0.0.1:4003
2013/10/25 22:50:24 kid2| Shutdown: NTLM authentication.
2013/10/25 22:50:24 kid2| Shutdown: Negotiate authentication.
2013/10/25 22:50:24 kid3| Shutdown: NTLM authentication.
2013/10/25 22:50:24 kid3| Shutdown: Negotiate authentication.
2013/10/25 22:50:24 kid2|

[squid-users] Re: Optimized Configuration?

[Linda Walsh]

Eliezer Croitoru wrote:

The basics...
Good hardware..
If you have good hardware there is nothing much you need to tune.
How many users?

anyone had the chance to do a MIPS vs INTEL GB lan card?


More pertinent, today would be a 10GB comparison.

My systems regularly pegged a 1GB Intel card in sys-sys file copies
(125MB/s writes, 119MB/s reads)...

Best speeds I've gotten off of 2x10GB-has been in the 700MB range,
though typical is more likely in the 200-400MB range.  HW/SW -- through
file server to null files tops out due to HW on both ends
(both Xeon Nehalams, one 12core@2.6 other 6core at 3.4).

(Intel cards between them direct connect.

[squid-users] what is lib.a in src/repl?

[Linda Walsh]

I just "upgraded(?)" some libraries / rpm from my 11.4->12.1 open suse 
release and

find I am no longer able to build squid --

I'm getting a weird error that indicates "something" is missing, but it is
rather generic...

The other libs in the dir are "libheap" and "lru"...

Looks like my _repl_acement policies... I was using mem: heap GDSF
and cache replacement heap LFUDA

Don't see how LFUDA or GDSF tie to the heap policy... or why either would
map to a generic name like lib.a?

I did re-run 'configure'...hmmm...maybe I need to rebootstrap?

weird...(3.2.0.17)...



libtool: link: /usr/bin/ar cru .libs/libfs.a .libs/Module.o
libtool: link: ranlib .libs/libfs.a
libtool: link: ( cd ".libs" && rm -f "libfs.la" && ln -s "../libfs.la" 
"libfs.la" )

make[3]: Leaving directory `/home/tools/squid/squid-3.2.0.17/src/fs'
Making all in 
repl  
<-

make[3]: Entering directory `/home/tools/squid/squid-3.2.0.17/src/repl'
make[3]: *** No rule to make target `lib.a', needed by `all-am'.  
Stop.   <-

make[3]: Leaving directory `/home/tools/squid/squid-3.2.0.17/src/repl'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/tools/squid/squid-3.2.0.17/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/tools/squid/squid-3.2.0.17/src'
make: *** [all-recursive] Error 1

[squid-users] Re: Correctoions (was TCP_SWAPFAIL/200)

[Linda Walsh]

Amos Jeffries wrote:


On 18.04.2012 12:46, Linda Walsh wrote:




It appears the local disk-store isn't growing over time -- so I'm
assuming it it telling
me the on-disk store isn't working right?


Yes.





Please prioritise the core dump investigation.
Please use gdb and find out what the crash is coming from. The crash and 
core-dump could be what is behind those incomplete or truncated responses.



At this point I suggest updating to 3.2.0.17. There are a bunch of cache 
related fixes in that release. The new cache swap.state format will 
rebuild your cache_dir meta data from scratch and discard anything which 
has problems visible.



---
I prioritized upgrading to the latest release and will go from there
(no need waiting time in things that may have been fixed).




If the core dumps continue with the new release, please prioritise 
those. Most of the rest of what you describe may be side effects of the 
crashing.



---
Will do...



http_access allow CONNECT Safe_Ports


NOTE: Dangerous. Safe_Ports includes port 1024-65535 and other ports 
unsafe to permit CONNECT to. This could trivially be used as a 
multi-stage spam proxy or worse.
  ie a trivial DoS of "CONNECT localhost:8080 HTTP/1.1\n\n" results in 
CONNECT loop until your machines port are all used up.



Good point, Just wanted to allow the general case of SSL/non-SSL over any of the
ports.  Just tryig to get things working at this point... though have had his 
config for soem time and no probs -- only connector is on my side and 'me', so

I shouldn't deny myself my own service unless I try!  ;-)





hierarchy_stoplist cgi-bin ?


You can drop hierarchy_stoplist from your config for simplicity.


---
check (some of these are carry-overs from prev configs or designed should
I use it for a different config).



cache_mem   8 GB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/cache/squid 65535 64 64


You have multiple workers configured. AUFS does not support SMP at this 
time. That could be the problem you have with SWAPFAIL, as the workers 
collide altering the cache contents.


---
Wah?   .. but but...how do I make use of SMP with AUFS?

If I go with uniq cache dirs that's very sub-optimal -- since I end up
with 12 separate cache areas, no?  when I want to fetch something from
the catch is there coordination about what content is in which worker's cache
that will automatically invoke the correct worker?   -- If so, that's cool,
but if not, then I'll reduce my hit rate by 1/N-cpus





To use this cache either wrap it in "if ${process_number} = N" tests for 
the workers you want to do caching. Or add ${process_number} to the path 
for each worker to get its own unique directory area.


eg:
 cache_dir aufs /var/cache/squid_${process_number} 65535 64 64

or
if ${process_number} = 1
 cache_dir aufs /var/cache/squid 65535 64 64
endif





--- As said above, how do I get multi-benefit with asynchronous writes
and multi core?







url_rewrite_host_header off
url_rewrite_access deny all
url_rewrite_bypass on


You do not have any re-writer or redirector configured. These 
url_rewrite_* can all go.


-
Is it harmful (it was for future 'expansion plans' -- no
rewriters yet, but was planning...)





refresh_pattern -i (/cgi-bin/|\?) 0 0%  0


This above pattern ...




 above what pattern?




refresh_pattern -i \.(ico|gif|jpg|png)   0 20%   4320
ignore-no-cache ignore-private override-expire
refresh_pattern -i ^http:   0 20%   4320ignore-no-cache 
ignore-private


"private" means the contents MUST NOT be served to multiple clients. 
Since you say this is a personal proxy just for you, thats okay but be 
carefulif you ever open it for use by other people. Things like your 
personal details embeded in same pages are cached by this.



Got it... I should add a comment in that area to that effect


That might be a enhancement -- like -
ignore-private-same-client




"no-cache" *actually* just means check for updates before using the 
cached version. This is usually not as useful as many tutorials make it 
out to be.


---
Well, dang tutorials -- I'm screwed if I follow, and if I don't! ;-)







refresh_pattern ^ftp:   144020% 10080
refresh_pattern ^gopher:14400%  1440


 ... is meant to be here (second to last).


refresh_pattern .   0   20% 4320
read_ahead_gap 256 MB


Uhm... 256 MB buffering per request sure you want to do that?



I **think*** so... doesn't that mean it will buffer up to 256MB
of a request before my client is read for it?

I think of the common case where I am saving a file and it takes me
a while to find the dir to save to.  I tweaked a few params in this area,
and it went from having to wai

[squid-users] Re: Is Adobe Connect Pro possible through Squid?

[Linda Walsh]

Linda Walsh wrote:


Peter Olsson wrote:


Hello!

Squid 3.1.19.

Users behind squid can't connect to Adobe Connect Pro,
probably because of RTMP. I have port 1935 in SSL_ports
and 1025-65535 in Safe_ports.

Is there anyone who had success connecting with
Adobe Connect Pro through squid?

Thanks!






I can connect to their test site... through my squid...
http://na1cps.adobeconnect.com/common/help/en/support/meeting_test.htm

I got "connection is LAN speed" through squid, (over a 16/2.5 Cable modem)...

[squid-users] Re: Is Adobe Connect Pro possible through Squid?

[Linda Walsh]

Peter Olsson wrote:


Hello!

Squid 3.1.19.

Users behind squid can't connect to Adobe Connect Pro,
probably because of RTMP. I have port 1935 in SSL_ports
and 1025-65535 in Safe_ports.

Is there anyone who had success connecting with
Adobe Connect Pro through squid?

Thanks!



From their requirements page, it doesn't seem like it would
preclude a proxy...

But you might have to use a transparent mode proxy for it to work...
i.e. port forwarding...

Is there somewhere in the client where you can specify a proxy port?

If so, I'd bet RTMP would work through a CONNECT session, but if there
is no place to plug in a proxy -- then it's probably trying to go direct.

Have you run a wireshark trace to see if it is trying to get through your
proxy or trying to go direct?

wireshark is very useful to get to the bottom of these things and you don't
have to be a network expert to use it... you'll see if it is talking to your
proxy port or if an HTTP request is trying to go direct to the net (not through
a proxy)...



Says, port requirements:

1935 (RTMP), 80 or other HTTP port, 443 if SSL is enabled, 25 for SMTP 
(optional), 1433 for external database (optional)


I've found SNAT or masquerading to be a requirement with some apps...then you
have to config squid as a transparent proxy (which i've never done, so don't 
know how hard that is)...I'm sure someone who has done it would say it is not

hard, but once you know how to do something, it's rarely 'hard'...;-)

[squid-users] TCP_SWAPFAIL/200

[Linda Walsh]

I recently (well a month or so ago) tried to upgrade squid after my old 
version got overwritten
by an OS related upgrade. 

now I am seeing TCP_SWAPFAIL/200 messages in my log -- that doesn't 
sound good.


Why would I be getting such?

It appears the local disk-store isn't growing over time -- so I'm 
assuming it it telling

me the on-disk store isn't working right?

I used a similar config to my previous one (below), so I'm not sure why 
it would be
croaking now...  Is there something "illegal" bout my config? 
I also included my non-comment squid.conf lines following that just to be

thorough
i'd really like to get squid back to being 100% 
solid-bullet-proof...which it

isn't right now (have had truncated downloads on longer downloads)...

I'm also getting occasional core dumps in the base of the cache dir, 
which is
usually a bad sign... ;-|  Haven't had a chance to try to check the 
stack trace yet,

but was wondering if anything looked amiss with my swap setup.


It's a 12core 48G machine, with a reasonably fast Raid so it should
have plenty of horse power for 1 user...but I find it can't keep up with
my browsing habits... which is insane considering it's usually used
for 10's-100's of users w/no prob...I know I am not that fast..


Any pointing out of "gotcha's" would be appreciated!...



squid -v
Squid Cache: Version 3.2.0.16
configure options:  'CFLAGS=-g -m64 -O2 -march=native -pipe -D_REENTRENT
'CCFLAGS=-g -m64 -O2 -march=native -pipe -D_REENTRENT 'LDFLAGS= -s'
'--prefix=/usr' '--bindir=/usr/sbin' '--datadir=/usr/share/squid'
'--libexecdir=/usr/sbin' '--libdir=/usr/lib64'
'--localstatedir=/var/cache/squid' '--sharedstatedir=/var/lib/squid'
'--sysconfdir=/etc/squid' '--docdir=/usr/share/packages/doc/squid'
'--with-aufs-threads=24' '--with-logdir=/var/log/squid'
'--with-mandir=/usr/share/man' '--with-piddir=/var/run/squid/squid.pid'
'--with-default-user=squid' '--with-gnu-ld' '--with-included-ltdl'
'--with-pic' '--with-large '--with-ltdl-lib=/usr/lib64'
'--enable-build-info' '--enable-cachemgr-hostname' '--enable-disk-io'
'--disable-ecap' '--disable-icap-client' '--enable-kill-parent-hack'
'--enable-linux-netfilter' '--enable-ltld-install' '--enable-referer-log'
'--enable-removal-policies' '--enable-stacktraces' '--enable-storeio'
'--enable-useragent-log' '--enable-zph-qos' '--enable-x-accelerator-vary'
'--disable-xmalloc-statistics' '--disable-auto-locale' '--disable-htcp'
'--disable-ident-lookups' '--disable-ipv6' '--disable-snmp'
'--disable-translation' '--without-netfilter-conntrack'
'EXT_LIBECAP_CFLAGS=-lecap' 'EXT_LIBECAP_LIBS=/usr/lib/libecap.so.2'

+ a bunch of compiler optimization switches: (that I also mostly used,
 though gcc is a newer version and a few options might be different)

-fpie -fmessage-length=0 -funwind-tables -fasynchronous-unwind-tables
-fbranch-target-load-optimize -fira-loop-pressure -fgcse -fgcse-las
-fgcse-lm -fgcse-sm -floop-interchange -floop-strip-mine -floop-block 
-flto -fpredictive-commoning -frename-registers -ftree-loop-linear 
-ftracer -ftree-loop-distribution -ftree-loop-im -ftree-loop-ivcanon 
-fivopts -ftree-vectorize -funswitch-loops 
-fvariable-expansion-in-unroller -freorder-blocks-and-partition -fweb'




Non-comment squid conf lines :

acl sc_subnet src 192.168.3.0/24
acl localnet src 192.168.3.0/24 # RFC1918 possible internal network
acl localnet src fc00::/7   # RFC 4193 local private network range
acl localnet src fe80::/10  # RFC 4291 link-local (directly plugged) 
machines

acl SSL_ports port 443
acl Safe_ports port 80  # http
acl Safe_ports port 81  # http
acl Safe_ports port 82  # http
acl Safe_ports port 21  # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70  # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1024-65535  # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Allowed_Connect port 1024-65535 #allowed non-SSL Connects to 
non-reserved ports

acl CONNECT method CONNECT
http_access allow manager localhost
http_access allow manager sc_subnet
http_access deny manager
http_access deny !Safe_ports
http_access allow CONNECT Safe_Ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 192.168.3.1:8080
hierarchy_stoplist cgi-bin ?
cache_mem   8 GB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/cache/squid 65535 64 64
maximum_object_size 1 GB
cache_swap_low 93
cache_store_log /var/log/squid/store.log
pid_filename /var/run/squid/squid.pid
strip_query_terms off
buffered_logs on
cache_log daemon:/var/log/squid/cache.log
coredump_dir /var/cache/squid
url_rewrite_host_header off
url_rewrite_access deny all
url_rewrite_bypass on
refresh_pattern

[squid-users] FWD: squid 3.2.0.16 access log no longer strictly increase...? *ouch* -- bug or feature?

[Linda Walsh]

repost -- mailer bounded it -- for some reason it thinks that having multiple 
formats of a message to choose from is a security problem?  Gee...

good think squid doesn't reject things with type-info, or it wouldn't work at 
all.

Guess things that reject all types are becoming the dinosaurs of the net.


To: squid-users@squid-cache.org
Subject: squid 3.2.0.16 access log no longer strictly  increase...?  *ouch* -- 
bug or feature?

Content-Type: multipart/alternative;
 boundary="090103050306030004050800"

This is a multi-part message in MIME format.
--090103050306030004050800
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit


Got a surprise in a new version of squid 3.2.0.16 -- in monitoring my
log, my monitor prog burped and died.  It didn't like trying to
calculate the average rate over negative time period (for some reason it
doesn't realize that time
can run backwards and data is actually sucked back out...;-))

Is this normal now?  One of the things I changed recently was going from
syslog, which tends to be pretty good about not having times go backwards,
to 'diskd'

Dunno if it is a bug or a feature, but it is a bit odd looking in a
time-progression based log.  I can at least prevent my script from gagging
at such, but is it supposed to be doing that??

Thanks -- sample times included below from log  -- ALL of these were
while I was connected to 'google', so ALL of them were 'CONNECT' log
messages, which might enter into the equation somewhere...
-linda
(** indicates time going backwards)

1332885640.662
1332885641.442
1332885641.431**
1332885641.436
1332885644.023
1332885663.344**
1332885663.515
...
1332885668.461
1332885670.637
1332885671.136
1332885672.015
1332885671.807**
1332885672.527
1332885672.325**
-

p.s. -- OTOH, it could be a marketing feature... squid's so fast, it
fetches content in negative time!




--090103050306030004050800
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit




  
  
/* Linda's Style playground (c) 2011 L. A Walsh (permission given to
   do w/this anything other than claim my original as your own!