Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Alexandre Chappaz]

Hi,

I added a loop waiting for the end of all squid processes ( with 30
seconds limit, I don't want to loop forever ..) and it did the trick :

for i in {1..30}
do
sleep 1
pidof 'squid'  /dev/null
pssquid=$?
if [ $pssquid -eq 0 ];then
echo Attente fin de process squid-z   /var/tmp/demarrage-squid.txt
else
echo tous les process sont terminés  /var/tmp/demarrage-squid.txt
break
fi
done


Thanks for your input.

Maybe this info has it's place here :
http://wiki.squid-cache.org/Features/SmpScale#Troubleshooting


Regards
Alex

2013/11/21 Alexandre Chappaz alexandrechap...@gmail.com:
 Thanks and yes this is exactly what we are doing.
 I will modify the init script so that it waits for the effective end of the
 squid-z before starting the daemon.

 Le 20 nov. 2013 18:16, Alex Rousskov rouss...@measurement-factory.com a
 écrit :

 On 11/20/2013 02:19 AM, Alexandre Chappaz wrote:

  I have the same kind of error but what bugs me is that I cannot
  reproduce this systematically. I am really wondering if this is a
  permission PB on shm mount point and / or  /var/run/squid permissions
  :
 
  some times the service starts normally ( worker kids stay up ) and
  some times some or all of the the worker kids die with this error :
 
  FATAL: Ipc::Mem::Segment::open failed to
  shm_open(/squid-cache_mem.shm): (2) No such file or directory.


 This is usually caused by two SMP Squid instances running, which is
 usually caused by incorrect squid -z application in the system
 startup/service scripts. YMMV, but the logs you posted later seem to
 suggest that it is exactly what is happening in your case.

 Do you run squid-z from the system startup/service script? If yes, does
 the script assume that squid -z ends when the squid -z command returns?
 If yes, the script should be modified to avoid that assumption because,
 in recent Squid releases, the squid-z instance continues to run (in the
 background) and clash with the regular squid instance started by the
 same script a moment later.

 There was a recent squid-dev discussion about fixing squid-z. I am not
 sure there was a strong consensus regarding the best solution, but I
 hope that squid-z will start doing nothing (Squid will just exit with a
 warning message about the deprecated option) in the foreseeable future
 while Squid instances will be capable of creating missing directories
 runtime, when needed (and allowed) to do so.

 More details and a call for volunteers at
  http://www.squid-cache.org/mail-archive/squid-dev/201311/0017.html


 HTH,

 Alex.

Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Alexandre Chappaz]

Hi,

I have the same kind of error but what bugs me is that I cannot
reproduce this systematically. I am really wondering if this is a
permission PB on shm mount point and / or  /var/run/squid permissions
:

some times the service starts normally ( worker kids stay up ) and
some times some or all of the the worker kids die with this error :

FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squid-cache_mem.shm): (2) No such file or directory.



attached is the cache.log, and here below the squid.conf.

Best regards


# pour le debogage (ne pas mettre plus de 2)
#debug_options ALL,2

# Utilisateurs
cache_effective_user nobody
cache_effective_group nobody


# Format access.log
strip_query_terms off
#logformat Squid  %ts.%03tu %6tr %a %Ss/%Hs %st %rm %ru %un %Sh/%A %mt
logformat PAS-Bdx %ts.%03tu %6tr %a %Ss/%Hs %st %rm %ru %un %Sh/%A
%mt %rv %tl %{Referer}h %{User-Agent}h

# chemins
coredump_dir /var/cache/squid
pid_filename /var/run/squid/squid.pid
access_log stdio:/var/log/squid/access.log PAS-Bdx
cache_log /var/log/squid/cache.log
cache_store_log none
mime_table /etc/squid/mime.conf
error_directory /etc/squid/errors
error_default_language fr
err_page_stylesheet /etc/squid/errorpage.css

# Fichier hosts
hosts_file /etc/hosts

# SNMP
acl snmpcommunity snmp_community read_only_user
snmp_access allow snmpcommunity
snmp_port 3401

###
# FONCTIONNEMENT DU PROXY #
###

#SMP
workers 4

#Ports d'ecoute
http_port 3128

#localhost a droit au cachemanager
http_access allow localhost manager
http_access deny manager

#localhost a droit a purger le cache
acl PURGE method PURGE
http_access allow PURGE localhost
http_access deny PURGE

# Les requetes intranet sont retournees en erreur
acl ip_intranet dst 10.0.0.0/8
http_access deny ip_intranet


acl PLSU_SIE_USERAGENT browser PLSU_SIE
acl PLSU_SIE_DEST dstdomain /etc/squid/acl/dest/PLSU_SIE.dst

http_access allow PLSU_SIE_USERAGENT PLSU_SIE_DEST
http_access deny PLSU_SIE_USERAGENT

#définition de la VIP des squid Père
#cache_peer 192.168.1.129 parent 3128 0 default no-query no-digest
cache_peer 192.168.1.201 parent 3128 0 sourcehash no-query no-digest
cache_peer 192.168.1.202 parent 3128 0 sourcehash no-query no-digest
cache_peer 192.168.1.203 parent 3128 0 sourcehash no-query no-digest
cache_peer 192.168.1.204 parent 3128 0 sourcehash no-query no-digest


# Time Out / Time To Live
negative_ttl 1 seconds
read_timeout 15 minutes
request_timeout 5 minutes
client_lifetime 4 hours
positive_dns_ttl 2 hours
negative_dns_ttl 5 minutes
shutdown_lifetime 5 seconds
dns_nameservers 127.0.0.1

# Divers
ftp_passive on
ftp_epsv off
logfile_rotate 2
request_header_access Via deny all
request_header_access X-Forwarded-For allow all
refresh_all_ims on

###
# FONCTIONNEMENT DU CACHE #
###

#Rafraichissement du cache
memory_cache_shared on
cache_mem 2 GB
max_filedesc 65535
maximum_object_size 512 MB
maximum_object_size_in_memory 2048 KB
ipcache_size 8192
fqdncache_size 8192

#definition du cache
#8Gb of shared rock cache, for 32Ko objects max
cache_dir rock /var/cache/squid/mem/ 8192 max-size=32768

if ${process_number} =1
# Filtrage avec squidGuard
url_rewrite_program /usr/local/squidGuard/bin/squidGuard
url_rewrite_children 1000 startup=15 idle=15 concurrency=0
cache_dir aufs /var/cache/squid/mem/W${process_number} 2048 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 12000 16 256 min-size=131072
endif
if ${process_number} =2
# Filtrage avec squidGuard
url_rewrite_program /usr/local/squidGuard/bin/squidGuard
url_rewrite_children 1000 startup=15 idle=15 concurrency=0
cache_dir aufs /var/cache/squid/mem/W${process_number} 2048 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 12000 16 256 min-size=131072
endif
if ${process_number} =3
# Filtrage avec squidGuard
url_rewrite_program /usr/local/squidGuard/bin/squidGuard
url_rewrite_children 1000 startup=15 idle=15 concurrency=0
cache_dir aufs /var/cache/squid/mem/W${process_number} 2048 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 12000 16 256 min-size=131072
endif
if ${process_number} =4
# Filtrage avec squidGuard
url_rewrite_program /usr/local/squidGuard/bin/squidGuard
url_rewrite_children 1000 startup=15 idle=15 concurrency=0
cache_dir aufs /var/cache/squid/mem/W${process_number} 2048 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 12000 16 256 min-size=131072
endif

# pages dynamiques non mises en cache
acl QUERY urlpath_regex cgi-bin \? \.fcgi \.cgi \.pl \.php3 \.asp \.php \.do
no_cache deny QUERY

# Reecriture des regles de gestion du cache pour certains domaines
gros consommateurs
acl forcedcache urlpath_regex .lefigaro\.fr .leparisien\.fr
.20minutes\.fr .lemde\.fr .lemonde\.fr .lepoint\.fr .lexpress\.fr
.meteofrance\.com .ouest-france\.fr .nouvelobs\.com .wikimedia\.org

Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Alexandre Chappaz]

here it is

2013/11/20 Eliezer Croitoru elie...@ngtech.co.il:
 Hey Alexandre,

 I do not see any cache.log attachment here.
 Please resend it.

 Thanks,
 Eliezer


 On 20/11/13 11:19, Alexandre Chappaz wrote:

 Hi,

 I have the same kind of error but what bugs me is that I cannot
 reproduce this systematically. I am really wondering if this is a
 permission PB on shm mount point and / or  /var/run/squid permissions
 :

 some times the service starts normally ( worker kids stay up ) and
 some times some or all of the the worker kids die with this error :

 FATAL: Ipc::Mem::Segment::open failed to
 shm_open(/squid-cache_mem.shm): (2) No such file or directory.



 attached is the cache.log, and here below the squid.conf.

 Best regards




cache.log.bz2
Description: BZip2 compressed data

Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Alex Rousskov]

On 11/20/2013 02:19 AM, Alexandre Chappaz wrote:

 I have the same kind of error but what bugs me is that I cannot
 reproduce this systematically. I am really wondering if this is a
 permission PB on shm mount point and / or  /var/run/squid permissions
 :
 
 some times the service starts normally ( worker kids stay up ) and
 some times some or all of the the worker kids die with this error :
 
 FATAL: Ipc::Mem::Segment::open failed to
 shm_open(/squid-cache_mem.shm): (2) No such file or directory.


This is usually caused by two SMP Squid instances running, which is
usually caused by incorrect squid -z application in the system
startup/service scripts. YMMV, but the logs you posted later seem to
suggest that it is exactly what is happening in your case.

Do you run squid-z from the system startup/service script? If yes, does
the script assume that squid -z ends when the squid -z command returns?
If yes, the script should be modified to avoid that assumption because,
in recent Squid releases, the squid-z instance continues to run (in the
background) and clash with the regular squid instance started by the
same script a moment later.

There was a recent squid-dev discussion about fixing squid-z. I am not
sure there was a strong consensus regarding the best solution, but I
hope that squid-z will start doing nothing (Squid will just exit with a
warning message about the deprecated option) in the foreseeable future
while Squid instances will be capable of creating missing directories
runtime, when needed (and allowed) to do so.

More details and a call for volunteers at
 http://www.squid-cache.org/mail-archive/squid-dev/201311/0017.html


HTH,

Alex.

Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Ricardo Klein]

Amos,

in some way I did something wrong with those permissions (I checked
them before post here, but, dont know why I didnt saw that they where
wrong).

Anyway, working 3.4.0.2 on Slackware 14.1 (rc2) with 2 workers and
rock storage.
Next test will be with CentOS 6.4 + NTLM authentication and LDAP_group
helper (we set permissions based on where the user is in AD groups,
so, support people can change users access permissions without asking
us).


--
Att...

Ricardo Felipe Klein
klein@gmail.com


On Sat, Oct 26, 2013 at 12:25 AM, Amos Jeffries squ...@treenet.co.nz wrote:
 On 26/10/2013 1:13 p.m., Ricardo Klein wrote:

 I am trying to run latest squid (for test purposes) and even on 3.3.9
 I always get:
 Squid Cache (Version 3.4.0.2): Terminated abnormally.
 CPU Usage: 0.015 seconds = 0.012 user + 0.003 sys
 Maximum Resident Size: 24864 KB
 Page faults with physical i/o: 0
 FATAL: Ipc::Mem::Segment::open failed to
 shm_open(/squid-squid-page-pool.shm): (2) No such file or directory

 Anyone Know why?


 The SHM socket/pipe for SMP worker communications cannot be opened by Squid.

 Check the permissions of /var/sun/squid.

 NP: if you are using MacOS there is something strange about the OS not
 accepting the normal read/write flags needed to open it.

 Amos

[squid-users] squid 3.4.0.2 + smp + rock storage error

[Ricardo Klein]

I am trying to run latest squid (for test purposes) and even on 3.3.9
I always get:
Squid Cache (Version 3.4.0.2): Terminated abnormally.
CPU Usage: 0.015 seconds = 0.012 user + 0.003 sys
Maximum Resident Size: 24864 KB
Page faults with physical i/o: 0
FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squid-squid-page-pool.shm): (2) No such file or directory

Anyone Know why?
Here is my squid.conf:
#SQUID.CONF#
visible_hostname squid.domain.local

cache_effective_usersquid
cache_effective_groupsquid
pid_filename/var/run/squid/squid.pid

ipcache_size 4096
ipcache_low 90
ipcache_high 95

fqdncache_size 4096

forward_max_tries 30

cache_mem 256 MB

cache_dir  rock /var/spool/squid/cache1 524288 max-size=1024
workers 2

cache_replacement_policy heap LFUDA

#ACLs
acl localnet src 192.168.2.0/24# RFC1918 possible internal network

acl SSL_ports port 443
acl SSL_ports port 5222
acl Safe_ports port 80# http
acl Safe_ports port 8080# http
acl Safe_ports port 21# ftp
acl Safe_ports port 443# https
acl Safe_ports port 70# gopher
acl Safe_ports port 210# wais
acl Safe_ports port 1025-65535# unregistered ports
acl Safe_ports port 280# http-mgmt
acl Safe_ports port 488# gss-http
acl Safe_ports port 591# filemaker
acl Safe_ports port 777# multiling http
acl Safe_ports port 5222
acl CONNECT method CONNECT

# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Just Allow everything
http_access allow localnet

# Squid normally listens to port 3128
http_port 192.168.2.2:3128

# We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid
#SQUID.CONF#

And my configure options (I know I have buit more then I need, but,
tests will grow to use ldap, ntlm and etc...):
Squid Cache: Version 3.4.0.2
configure options:  '--prefix=/usr' '--libdir=/usr/lib64'
'--sysconfdir=/etc/squid' '--localstatedir=/var/log/squid'
'--datadir=/usr/share/squid' '--with-pidfile=/var/run/squid'
'--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/'
'--mandir=/usr/man' '--enable-snmp' '--enable-follow-x-forwarded-for'
'--enable-auth-basic=LDAP,MSNT,MSNT-multi-domain,NCSA,SMB,getpwnam,SASL,RADIUS'
'--enable-auth-ntlm=smb_lm,fake,SSPI'
'--enable-auth-digest=file,LDAP,eDirectory'
'--enable-auth-negotiate=kerberos'
'--enable-external-acl-helpers=file_userip,LDAP_group,kerberos_ldap_group,session,unix_group,wbinfo_group'
'--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
'--enable-delay-pools' '--enable-epoll' '--enable-icap-client'
'--enable-ident-lookups' '--enable-linux-netfilter' '--enable-eui'
'--enable-async-io' '--build=x86_64-slackware-linux'
'--disable-strict-error-checking' '--disable-dependency-tracking'
'--enable-removal-policies=heap,lru' '--enable-storeio=rock'
'--enable-ssl' '--with-large-files' '--enable-referer-log'
'--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio'
'--with-default-user=squid' '--with-filedescriptors=30720' '--with-dl'
'--with-openssl' '--disable-ipv6' '--with-pthreads'
'build_alias=x86_64-slackware-linux' 'CFLAGS=-O2 -fPIC' 'CXXFLAGS=-O2
-fPIC' 'PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/lib64/pkgconfig'


--
Att...

Ricardo Felipe Klein
klein@gmail.com

Re: [squid-users] squid 3.4.0.2 + smp + rock storage error

[Amos Jeffries]

On 26/10/2013 1:13 p.m., Ricardo Klein wrote:

I am trying to run latest squid (for test purposes) and even on 3.3.9
I always get:
Squid Cache (Version 3.4.0.2): Terminated abnormally.
CPU Usage: 0.015 seconds = 0.012 user + 0.003 sys
Maximum Resident Size: 24864 KB
Page faults with physical i/o: 0
FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squid-squid-page-pool.shm): (2) No such file or directory

Anyone Know why?


The SHM socket/pipe for SMP worker communications cannot be opened by Squid.

Check the permissions of /var/sun/squid.

NP: if you are using MacOS there is something strange about the OS not 
accepting the normal read/write flags needed to open it.


Amos