[squid-users] Re: Youtube upload cpu problem
nobody? Em 20 de outubro de 2010 11:10, Marcelo Grassi F. Melgaço mgfra...@gmail.com escreveu: Greetings! I have a Firewall with Ubuntu 9.10 Server + Squid 3.0.STABLE18 (apt-get) My problem is, when a user start a upload in youtube, the cpu used by squid process go to 100% !! The others requests goes slow. I don't know whats is causing this. I have authentication with basic scheme and just some blocking acl for some domains. I have try to put the domain upload.youtube.com in no_cache, no sucess. Anyone can help me? After searching google i found nothing about it. Thanks a lot!
Re: [squid-users] Youtube upload cpu problem
As far as I know, that version is not recommended. Please use the latest stable versions and see if problem persists. Regards HASSAN On 2010-10-22, Marcelo Grassi F. Melgaço mgfra...@gmail.com wrote: nobody? Em 20 de outubro de 2010 11:10, Marcelo Grassi F. Melgaço mgfra...@gmail.com escreveu: Greetings! I have a Firewall with Ubuntu 9.10 Server + Squid 3.0.STABLE18 (apt-get) My problem is, when a user start a upload in youtube, the cpu used by squid process go to 100% !! The others requests goes slow. I don't know whats is causing this. I have authentication with basic scheme and just some blocking acl for some domains. I have try to put the domain upload.youtube.com in no_cache, no sucess. Anyone can help me? After searching google i found nothing about it. Thanks a lot! -- Sent from my mobile device
[squid-users] allowed sites acl gives problem
Dear All, I been using using for quite some time and itsa excellent stable product by the way I do have some diffculty I want to allow only specific sites to specific machines let me explain i have 3 machines with ip of 172.16.2.22, 172.16.2.23, 172.16.2.24 these three machine have to be able to have acceo only a few sites like www.yahoo.com, www.google.com and www.cnn.com and probably a couple will be added latter so i did add a acl like below acl sunray_allowed src 172.16.2.22 172.16.2.23 172.16.2.24 acl good_sites url_regex /etc/squid/allowed-sites.squid http_access allow sunray_allowed good_sites here is my allowed-sites.squid file .yahoo.com .google.com .cnn.com now when i go to www.google.com it works fine but when i go to yahoo or cnn the page is not displayed properly the squid access.log says - 287745303.890 0 172.16.2.23 TCP_DENIED/403 1311 GET http://i.cdn.turner.com/cnn/.element/js/3.0/s_code.js - NONE/- text/html 1287745303.903 0 172.16.2.23 TCP_DENIED/403 1309 GET http://content.dl-rms.com/rms/mother/5721/nodetag.js - NONE/- text/html 1287745303.911 0 172.16.2.23 TCP_DENIED/403 1333 GET http://i.cdn.turner.com/cnn/.element/js/3.0/hpsectiontracking.js - NONE/- text/html 1287745303.916 0 172.16.2.23 TCP_DENIED/403 1285 GET http://i.cdn.turner.com/cnn/images/1.gif - NONE/- text/html 1287745303.917 0 172.16.2.23 TCP_DENIED/403 1275 GET http://js.revsci.net/gateway/gw.js? - NONE/- text/html 1287745303.917997 172.16.2.23 TCP_MISS/000 0 GET http://www.cnn.com/ght= - DIRECT/157.166.224.26 - 1287745304.086724 172.16.2.23 TCP_MISS/302 730 GET http://www.cnn.com/.element/img/3.0/1px.gif - DIRECT/157.166.226.25 text/html 1287745304.999913 172.16.2.23 TCP_REFRESH_HIT/304 426 GET http://edition.cnn.com/.element/img/3.0/1px.gif - DIRECT/157.166.224.45 image/gif 1287745305.346327 172.16.2.23 TCP_REFRESH_MISS/302 727 GET http://www.cnn.com/tools/search/cnncom.xml - DIRECT/157.166.226.25 text/html other sites are denied as normal which is perfect. i also tried usin dstdomain in place of url_regex but the same problem I would really apprecite if someone could help me regards simon -- Network ADMIN - KUWAIT MUNICIPALITY: -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: [squid-users] Youtube upload cpu problem
I'll try to update. Thanks 2010/10/22 Nyamul Hassan mnhas...@usa.net: As far as I know, that version is not recommended. Please use the latest stable versions and see if problem persists. Regards HASSAN On 2010-10-22, Marcelo Grassi F. Melgaço mgfra...@gmail.com wrote: nobody? Em 20 de outubro de 2010 11:10, Marcelo Grassi F. Melgaço mgfra...@gmail.com escreveu: Greetings! I have a Firewall with Ubuntu 9.10 Server + Squid 3.0.STABLE18 (apt-get) My problem is, when a user start a upload in youtube, the cpu used by squid process go to 100% !! The others requests goes slow. I don't know whats is causing this. I have authentication with basic scheme and just some blocking acl for some domains. I have try to put the domain upload.youtube.com in no_cache, no sucess. Anyone can help me? After searching google i found nothing about it. Thanks a lot! -- Sent from my mobile device
Re: [squid-users] Youtube upload cpu problem
On 23/10/10 00:07, Marcelo Grassi F. Melgaço wrote: I'll try to update. Thanks This may help the update: https://launchpad.net/~yadi/+archive/ppa Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] Youtube upload cpu problem
Thank you very much ! I'll test upload now and them report here. Thanks again ! 2010/10/22 Amos Jeffries squ...@treenet.co.nz: On 23/10/10 00:07, Marcelo Grassi F. Melgaço wrote: I'll try to update. Thanks This may help the update: https://launchpad.net/~yadi/+archive/ppa Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] Youtube upload cpu problem
I update squid for 3.1.8 using the repository above. But, when i made a upload for youtube, still have the same problem. CPU goes to 100% and other requests goes slow. The access log shows this many times: 1287751157.275 2059 192.168.10.86 TCP_MISS/200 440 GET http://upload.youtube.com/api/upload_feedback? - DIRECT/74.125.47.118 text/plain Thanks for any help. 2010/10/22 Amos Jeffries squ...@treenet.co.nz: On 23/10/10 00:07, Marcelo Grassi F. Melgaço wrote: I'll try to update. Thanks This may help the update: https://launchpad.net/~yadi/+archive/ppa Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] Youtube upload cpu problem
Is there a way to limit upload speed for domain upload.youtube.com in squid? Maybe this help to resolve my problem.. Thanks Em 22 de outubro de 2010 10:40, Marcelo Grassi F. Melgaço mgfra...@gmail.com escreveu: I update squid for 3.1.8 using the repository above. But, when i made a upload for youtube, still have the same problem. CPU goes to 100% and other requests goes slow. The access log shows this many times: 1287751157.275 2059 192.168.10.86 TCP_MISS/200 440 GET http://upload.youtube.com/api/upload_feedback? - DIRECT/74.125.47.118 text/plain Thanks for any help. 2010/10/22 Amos Jeffries squ...@treenet.co.nz: On 23/10/10 00:07, Marcelo Grassi F. Melgaço wrote: I'll try to update. Thanks This may help the update: https://launchpad.net/~yadi/+archive/ppa Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
[squid-users] Re: Problem with SQUID_KERB_LDAP
Hi guru's I try some weeks to configure my squid to auth with MS AD with squid_kerb_auth. As i understand squid_kerb_ldap is a new helper for ldap requests instead of squid_ldap_group, or am i wrong? My squid.conf look like this: auth_param negotiate program /usr/local/squid/libexec/squid_kerb_auth -d -s HTTP/proxyhostname.mydomain.com auth_param negotiate children 2 auth_param negotiate keep_alive on external_acl_type SQUID_KERB_LDAP ttl=3600 negative_ttl=3600 %LOGIN /usr/local/squid/libexec/squid_kerb_ldap -g ProxyUsersGroup_in_AD acl LDAP_GROUP_CHECK external SQUID_KERB_LDAP http_access allow LDAP_GROUP_CHECK But when i start squid i have two problems: 1. squid_kerb_ldap didn't start 2010/10/21 16:19:09| Starting Squid Cache version 3.1.8 for i686-pc-linux-gnu... 2010/10/21 16:19:09| Process ID 7648 2010/10/21 16:19:09| With 1024 file descriptors available 2010/10/21 16:19:09| Initializing IP Cache... 2010/10/21 16:19:09| DNS Socket created at [::], FD 8 2010/10/21 16:19:09| DNS Socket created at 0.0.0.0, FD 9 2010/10/21 16:19:09| Adding domain mydomain.com from /etc/resolv.conf 2010/10/21 16:19:09| Adding nameserver 192.168.1.28 from /etc/resolv.conf 2010/10/21 16:19:09| Adding nameserver 192.168.1.17 from /etc/resolv.conf 2010/10/21 16:19:09| helperOpenServers: Starting 2/2 'squid_kerb_auth' processes 2010/10/21 16:19:09| squid_kerb_auth: INFO: Starting version 1.0.5 2010/10/21 16:19:09| helperOpenServers: Starting 5/5 'squid_kerb_ldap' processes 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| squid_kerb_auth: INFO: Starting version 1.0.5 2010/10/21 16:19:09| Unlinkd pipe opened on FD 28 2010/10/21 16:19:09| Store logging disabled 2010/10/21 16:19:09| Swap maxSize 0 + 262144 KB, estimated 20164 objects 2010/10/21 16:19:09| Target number of buckets: 1008 2010/10/21 16:19:09| Using 8192 Store buckets 2010/10/21 16:19:09| Max Mem size: 262144 KB 2010/10/21 16:19:09| Max Swap size: 0 KB 2010/10/21 16:19:09| Using Least Load store dir selection 2010/10/21 16:19:09| chdir: /usr/local/squid/var/cache: (2) No such file or directory 2010/10/21 16:19:09| Current Directory is /etc/init.d 2010/10/21 16:19:09| Loaded Icons. 2010/10/21 16:19:09| Accepting HTTP connections at [::]:80, FD 29. 2010/10/21 16:19:09| HTCP Disabled. 2010/10/21 16:19:09| Squid modules loaded: 0 2010/10/21 16:19:09| Ready to serve requests. 2010/10/21 16:19:10| storeLateRelease: released 0 objects Here detailed log with debug level 9 2010/10/21 17:41:24.062| comm_openex: Attempt open socket for: [::1] 2010/10/21 17:41:24.062| comm_openex: Opened socket FD 16 : family=10, type=1, protocol=0 2010/10/21 17:41:24.062| comm_open: FD 16 is a new socket 2010/10/21 17:41:24.062| fd_open() FD 16 squid_kerb_ldap 2010/10/21 17:41:24.062| commBind: bind socket FD 16 to [::1] 2010/10/21 17:41:24.062| comm_openex: Attempt open socket for: [::1] 2010/10/21 17:41:24.062| comm_openex: Opened socket FD 17 : family=10, type=1, protocol=0 2010/10/21 17:41:24.062| comm_open: FD 17 is a new socket 2010/10/21 17:41:24.062| fd_open() FD 17 squid_kerb_ldap 2010/10/21 17:41:24.062| commBind: bind socket FD 17 to [::1] 2010/10/21 17:41:24.062| ipcCreate: prfd FD 17 2010/10/21 17:41:24.062| ipcCreate: pwfd FD 17 2010/10/21 17:41:24.062| ipcCreate: crfd FD 16 2010/10/21 17:41:24.062| ipcCreate: cwfd FD 16 2010/10/21 17:41:24.062| ipcCreate: FD 17 sockaddr [::1]:60649 2010/10/21 17:41:24.062| ipcCreate: FD 16 sockaddr [::1]:47055 2010/10/21 17:41:24.062| ipcCreate: FD 16 listening... 2010/10/21 17:41:24.062| comm_close: start closing FD 16 2010/10/21 17:41:24.062| The AsyncCall comm_close_start constructed, this=0x83b85c0 [call13] 2010/10/21 17:41:24.062| comm.cc(1611) will call comm_close_start(FD 16) [call13] 2010/10/21 17:41:24.062| comm.cc(1195) commSetTimeout: FD 16 timeout -1 2010/10/21 17:41:24.062| comm.cc(1206) commSetTimeout: FD 16 timeout -1 2010/10/21 17:41:24.062| commCallCloseHandlers: FD 16 2010/10/21 17:41:24.062| The AsyncCall comm_close_complete constructed, this=0x83b8600 [call14] 2010/10/21 17:41:24.062| comm.cc(1643) will call comm_close_complete(FD 16) [call14] 2010/10/21 17:41:24.062| comm_connect_addr: connecting socket 17 to [::1]:47055 (want family: 10) 2010/10/21 17:41:24.062| comm_connect_addr: sock=17, addrinfo( flags=4, family=10, socktype=1, protocol=6, addr=0x83b8668, addrlen =28 ) 2010/10/21 17:41:24.062| connect FD 17: (-1) (101) Network is unreachable 2010/10/21 17:41:24.062| connecting to: [::1]:47055 2010/10/21 17:41:24.062| comm_close: start closing FD 17 2010/10/21 17:41:24.062| The
[squid-users] squid performance
Dear team, I run a Squid Cache: Version 3.1.8. i have a problem when my client_http.requests = is more than 200/sec. pages doesn't browse but when the request are less than 200 i dont find any problem. i don't see any errors in /etc/var/squid/cache.log. my file descriptors is 32768. Please find my configuration below and do suggest me if i m any where wrong in my configuration. Thanks in advance. my h/w details is as fallows: CPU: 3.00 GHZ XEON processor RAM: 8 GB HDD: 148 GB * 2 SAS HDD my ulimint -n = 32768 File descriptor usage for squid: Maximum number of file descriptors: 32768 Largest file desc currently in use: 6064 Number of file desc currently in use: 5656 Files queued for open: 0 Available number of file descriptors: 27112 Reserved number of file descriptors: 100 Store Disk files open: 119 my squid.conf: ### Start of squid.conf #created by ANANTH# cache_effective_user squid cache_effective_group squid http_port 3128 transparent # httpd_accel_host virtual # httpd_accel_port 80 # httpd_accel_with_proxy on # httpd_accel_uses_host_header on # cache_dir aufs /var/spool/squid 16384 32 512 #--This has been inserted to check the cache-- #cache_dir ufs /var/spool/squid 16384 16 256 #cache_dir ufs /cache0/squid 16384 16 256 #cache_dir ufs /squid0/squid 16384 16 256 cache_dir aufs /squid1/squid 16384 32 512 #cache_dir /tmp null cache_access_log /var/log/squid/access.log cache_log /var/log/squid/cache.log cache_store_log none logfile_rotate 7 emulate_httpd_log on cache_mem 3 GB maximum_object_size_in_memory 256 KB memory_replacement_policy lru cache_replacement_policy lru maximum_object_size 64 MB hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY hosts_file /etc/hosts refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 40% 4320 ##Define your network below #acl mynetwork src 192.168.0.0/24 acl mynetwork src 192.168.106.0/24 # cbinetwork private acl mynetwork src 192.168.107.0/24 # cbinetwork private acl mynetwork src 192.168.110.0/24 # cbinetwork private acl mynetwork src 192.168.120.0/24 # cbinetwork private acl mynetwork src 192.168.121.0/24 # cbinetwork private acl mynetwork src 192.168.130.0/24 # cbinetwork private acl mynetwork src 192.168.150.0/24 # cbinetwork private acl mynetwork src 192.168.151.0/24 # cbinetwork private acl mynetwork src 192.168.160.0/24 # cbinetwork private acl mynetwork src 10.100.101.0/24 # cbinetwork private acl manager proto cache_object acl localhost src 127.0.0.1/32 acl localhost src ::1/128 acl to_localhost dst 127.0.0.0/8 acl to_localhost dst ::1/128 acl purge method PURGE acl CONNECT method CONNECT acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 1025-65535 #unregistered ports acl SSL_ports port 443 563 http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow mynetwork # http_access deny all http_reply_access allow all icp_access allow mynetwork # icp_access deny all visible_hostname proxy.cbinet.bi coredump_dir /squid1/squid # ###
[squid-users] Re: Problem with SQUID_KERB_LDAP
DmitrySh sbro...@inbox.lv wrote in message news:1287753284416-3007186.p...@n4.nabble.com... Hi guru's I try some weeks to configure my squid to auth with MS AD with squid_kerb_auth. As i understand squid_kerb_ldap is a new helper for ldap requests instead of squid_ldap_group, or am i wrong? Yes that is correct My squid.conf look like this: auth_param negotiate program /usr/local/squid/libexec/squid_kerb_auth -d -s HTTP/proxyhostname.mydomain.com auth_param negotiate children 2 auth_param negotiate keep_alive on external_acl_type SQUID_KERB_LDAP ttl=3600 negative_ttl=3600 %LOGIN /usr/local/squid/libexec/squid_kerb_ldap -g ProxyUsersGroup_in_AD acl LDAP_GROUP_CHECK external SQUID_KERB_LDAP http_access allow LDAP_GROUP_CHECK But when i start squid i have two problems: 1. squid_kerb_ldap didn't start 2010/10/21 16:19:09| Starting Squid Cache version 3.1.8 for i686-pc-linux-gnu... 2010/10/21 16:19:09| Process ID 7648 2010/10/21 16:19:09| With 1024 file descriptors available 2010/10/21 16:19:09| Initializing IP Cache... 2010/10/21 16:19:09| DNS Socket created at [::], FD 8 2010/10/21 16:19:09| DNS Socket created at 0.0.0.0, FD 9 2010/10/21 16:19:09| Adding domain mydomain.com from /etc/resolv.conf 2010/10/21 16:19:09| Adding nameserver 192.168.1.28 from /etc/resolv.conf 2010/10/21 16:19:09| Adding nameserver 192.168.1.17 from /etc/resolv.conf 2010/10/21 16:19:09| helperOpenServers: Starting 2/2 'squid_kerb_auth' processes 2010/10/21 16:19:09| squid_kerb_auth: INFO: Starting version 1.0.5 2010/10/21 16:19:09| helperOpenServers: Starting 5/5 'squid_kerb_ldap' processes 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. Does it rin on the command line ? Do you have any process controls (selinux, apparmor) enabled ? 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| WARNING: Cannot run '/usr/local/squid/libexec/squid_kerb_ldap' process. 2010/10/21 16:19:09| squid_kerb_auth: INFO: Starting version 1.0.5 2010/10/21 16:19:09| Unlinkd pipe opened on FD 28 2010/10/21 16:19:09| Store logging disabled 2010/10/21 16:19:09| Swap maxSize 0 + 262144 KB, estimated 20164 objects 2010/10/21 16:19:09| Target number of buckets: 1008 2010/10/21 16:19:09| Using 8192 Store buckets 2010/10/21 16:19:09| Max Mem size: 262144 KB 2010/10/21 16:19:09| Max Swap size: 0 KB 2010/10/21 16:19:09| Using Least Load store dir selection 2010/10/21 16:19:09| chdir: /usr/local/squid/var/cache: (2) No such file or directory 2010/10/21 16:19:09| Current Directory is /etc/init.d 2010/10/21 16:19:09| Loaded Icons. 2010/10/21 16:19:09| Accepting HTTP connections at [::]:80, FD 29. 2010/10/21 16:19:09| HTCP Disabled. 2010/10/21 16:19:09| Squid modules loaded: 0 2010/10/21 16:19:09| Ready to serve requests. 2010/10/21 16:19:10| storeLateRelease: released 0 objects Here detailed log with debug level 9 2010/10/21 17:41:24.062| comm_openex: Attempt open socket for: [::1] 2010/10/21 17:41:24.062| comm_openex: Opened socket FD 16 : family=10, type=1, protocol=0 2010/10/21 17:41:24.062| comm_open: FD 16 is a new socket 2010/10/21 17:41:24.062| fd_open() FD 16 squid_kerb_ldap 2010/10/21 17:41:24.062| commBind: bind socket FD 16 to [::1] 2010/10/21 17:41:24.062| comm_openex: Attempt open socket for: [::1] 2010/10/21 17:41:24.062| comm_openex: Opened socket FD 17 : family=10, type=1, protocol=0 2010/10/21 17:41:24.062| comm_open: FD 17 is a new socket 2010/10/21 17:41:24.062| fd_open() FD 17 squid_kerb_ldap 2010/10/21 17:41:24.062| commBind: bind socket FD 17 to [::1] 2010/10/21 17:41:24.062| ipcCreate: prfd FD 17 2010/10/21 17:41:24.062| ipcCreate: pwfd FD 17 2010/10/21 17:41:24.062| ipcCreate: crfd FD 16 2010/10/21 17:41:24.062| ipcCreate: cwfd FD 16 2010/10/21 17:41:24.062| ipcCreate: FD 17 sockaddr [::1]:60649 2010/10/21 17:41:24.062| ipcCreate: FD 16 sockaddr [::1]:47055 2010/10/21 17:41:24.062| ipcCreate: FD 16 listening... 2010/10/21 17:41:24.062| comm_close: start closing FD 16 2010/10/21 17:41:24.062| The AsyncCall comm_close_start constructed, this=0x83b85c0 [call13] 2010/10/21 17:41:24.062| comm.cc(1611) will call comm_close_start(FD 16) [call13] 2010/10/21 17:41:24.062| comm.cc(1195) commSetTimeout: FD 16 timeout -1 2010/10/21 17:41:24.062| comm.cc(1206) commSetTimeout: FD 16 timeout -1 2010/10/21 17:41:24.062| commCallCloseHandlers: FD 16 2010/10/21 17:41:24.062| The AsyncCall comm_close_complete constructed, this=0x83b8600 [call14] 2010/10/21 17:41:24.062| comm.cc(1643) will call comm_close_complete(FD 16) [call14] 2010/10/21 17:41:24.062| comm_connect_addr: connecting socket 17 to [::1]:47055 (want family: 10) 2010/10/21 17:41:24.062| comm_connect_addr: sock=17, addrinfo( flags=4, family=10, socktype=1, protocol=6,
[squid-users] Re: cache_dir aufs min-size
Hi, I was wondering if someone could shed a light on my previous unanswered email. Thank you. On Tue, Oct 19, 2010 at 22:10, Costin Gusa costi...@gmail.com wrote: Hi, I would like to limit the minimum object size in a cache dir For this I have setup the following line in squid.conf: cache_dir aufs /var/spool/squid 65536 16 256 min-size=4MB However looking at /var/spool/squid cache dir, I see smaller objects also: /var/spool/squid# ls -lah 00/00|grep K|head -n3 drwxr-x--- 2 proxy proxy 4.0K 2010-10-19 21:56 . drwxr-x--- 258 proxy proxy 8.0K 2010-10-19 20:31 .. -rw-r- 1 proxy proxy 73K 2010-10-19 20:35 I tried also with min-size=4194304 with no luck Is this an intended behaviour or am I misunderstanding this functionality? [...] ii squid-common 2.7.STABLE3-4.1lenny1 Internet object cache (WWW proxy c
Re: [squid-users] Missing username on logs when using c-icap
Hi Amos. Based on the information of the configuration o sent before do you think I am missing something or should I report it as a bug. Squid properly passes the user to c-icap but it seans to forget it is one authenticated request and dont set the username on the log. Regards Carlos. - Original Message - From: Carlos Xavier cbas...@connection.com.br To: squid-users@squid-cache.org Sent: Wednesday, October 20, 2010 2:25 AM Subject: Re: [squid-users] Missing username on logs when using c-icap Hi.tank you for the reply. This is my squid version ans how it was compiled: Squid Cache: Version 3.1.8 configure options: '--with-maxfd=8192' '--prefix=/usr' '--libdir=/usr/lib64' '--sysconfdir=/etc/squid' '--localstatedir=/var/log/squid' '--datadir=/usr/share/squid' '--mandir=/usr/man' '--with-pthreads' '--enable-follow-x-forwarded-for' '--enable-storeio=aufs ufs diskd' '--enable-removal-policies=lru heap' '--enable-delay-pools' '--enable-snmp' '--enable-icap-client' '--enable-auth=basic ntlm digest' '--enable-basic-auth-helpers=DB LDAP NCSA MSNT SMB getpwnam' '--enable-digest-auth-helpers=ldap password' '--enable-ntlm-auth-helpers=smb_lm' '--enable-external-acl-helpers=ip_user ldap_group unix_group wbinfo_group' '--enable-linux-netfilter' '--enable-async-io' '--build=x86_64-slackware-linux' 'build_alias=x86_64-slackware-linux' 'CFLAGS=-O2 -fPIC' 'CXXFLAGS=-O2 -fPIC' --with-squid=/tmp/SBo/squid-3.1.8 Here is it´s configuration: auth_param basic program /usr/libexec/ncsa_auth /etc/squid/passwd auth_param basic children 5 auth_param basic realm Squid proxy-caching: SEU ACESSO ESTA SENDO MONITORADO auth_param basic credentialsttl 2 hours auth_param basic casesensitive off acl manager proto cache_object acl localhost src 127.0.0.1/32 acl localhost src ::1/128 acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 acl to_localhost dst ::1/128 acl SSL_ports port 443 563 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT acl SEARCH method SEARCH acl FTP proto FTP ... # Local ACLs acl PURGE method PURGE acl password proxy_auth REQUIRED acl localnet src 172.31.0.0/24 ... acl MULTIMEDIA rep_mime_type -i ^(audio\/x-mpegurl|audio\/mpeg)$ acl MULTIMEDIA rep_mime_type -i ^(video\/flv|video\/x-flvs)$ acl MULTIMEDIA rep_mime_type -i ^(application\/x-shockwave-flash||application\/ogg)$ acl MULTIMEDIA rep_mime_type -i ^(audio\/ogg|video\/ogg)$ ... http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost http_access allow PURGE localhost http_access deny PURGE http_access allow localnet password http_access deny all http_port 3128 hierarchy_stoplist cgi-bin ? cache_dir ufs /var/cache/squid/ 1000 16 256 access_log /var/log/squid/access.log squid icap_log /var/log/squid/icap_access.log cache_store_log none logfile_rotate 0 pid_filename /var/run/squid/squid.pid cache_log /var/log/squid/cache.log coredump_dir /var/log/squid refresh_pattern ^ftp: 144020% 10080 refresh_pattern ^gopher:14400% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 icap_enable on icap_send_client_ip on icap_send_client_username on icap_client_username_header X-Authenticated-User icap_service service_av_req reqmod_precache bypass=1 icap://localhost:1344/srv_clamav icap_service service_av_resp respmod_precache bypass=1 icap://localhost:1344/srv_clamav icap_service service_url_check_req reqmod_precache bypass=1 icap://localhost:1344/url_check adaptation_access service_av_resp deny MULTIMEDIA adaptation_access service_av_resp allow all adaptation_service_chain REQ_CHAIN service_url_check_req service_av_req adaptation_access REQ_CHAIN deny MULTIMEDIA adaptation_access REQ_CHAIN allow all: The only change made from the normal operation mod to the use of c-icap was to add the icap configuration and services. Regards. Carlos Xavier. - Original Message - From: Amos Jeffries squ...@treenet.co.nz To: squid-users@squid-cache.org Sent: Wednesday, October 20, 2010 12:46 AM Subject: Re: [squid-users] Missing username on logs when using c-icap On Tue, 19 Oct 2010 15:45:56 -0200, Carlos Xavier cbas...@connection.com.br wrote: Hi. We use SARG to generate some statistical data and also to have some control where our authenticated users are going. This control are based on the username on the squid access log. Now we started to use c-icap to check for virus and check the url. Since then the username of the users doing the request
Re: [squid-users] squid performance
On 23/10/10 03:01, Ananth wrote: Dear team, I run a Squid Cache: Version 3.1.8. i have a problem when my client_http.requests = is more than 200/sec. pages doesn't browse but when the request are less than 200 i dont find any problem. i don't see any errors in /etc/var/squid/cache.log. my file descriptors is 32768. Please find my configuration below and do suggest me if i m any where wrong in my configuration. There is nothing visibly wrong with the below config. It's essentially the default one which most are using happily. I've pointed out a few bits which could be improved for overall speed, but the gains are not ones which would suddenly cut in like that. What does squid -v produce? and what OS is this on please? Thanks in advance. my h/w details is as fallows: CPU: 3.00 GHZ XEON processor RAM: 8 GB HDD: 148 GB * 2 SAS HDD my ulimint -n = 32768 File descriptor usage for squid: Maximum number of file descriptors: 32768 Largest file desc currently in use: 6064 Number of file desc currently in use: 5656 Files queued for open: 0 Available number of file descriptors: 27112 Reserved number of file descriptors: 100 Store Disk files open: 119 my squid.conf: ### Start of squid.conf #created by ANANTH# cache_effective_user squid cache_effective_group squid effective-group is a piece of major voodoo with VERY limited real use-cases. *general* recommendation is to trust the OS security settings membership of squid user and remove that group option from the config. http_port 3128 transparent With 3.1 this is now intercept to avoid confusion with tproxy (transparent proxy). # httpd_accel_host virtual # httpd_accel_port 80 # httpd_accel_with_proxy on # httpd_accel_uses_host_header on Um, those should be removed. From your choice of transparent as a replacement I'm assuming you want this as a transparent interception-proxy. If you want it as a reverse-proxy (what those old config lines did) that is a whole separate config now. # cache_dir aufs /var/spool/squid 16384 32 512 #--This has been inserted to check the cache-- #cache_dir ufs /var/spool/squid 16384 16 256 #cache_dir ufs /cache0/squid 16384 16 256 #cache_dir ufs /squid0/squid 16384 16 256 cache_dir aufs /squid1/squid 16384 32 512 #cache_dir /tmp null cache_access_log /var/log/squid/access.log cache_log /var/log/squid/cache.log cache_store_log none logfile_rotate 7 emulate_httpd_log on Drop emulate_httpd_log and cache_access_log. Use this instead for the same output slightly faster: access_log /var/log/squid/access.log common cache_mem 3 GB maximum_object_size_in_memory 256 KB memory_replacement_policy lru cache_replacement_policy lru maximum_object_size 64 MB hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY Drop the QUERY and cgi-bin stuff here. It will be forcing your Squid to do slow network fetches for a lot of otherwise cacheable dynamic pages. There is a refresh_pattern below which fixes up the non-cacheable ones behaviour. hosts_file /etc/hosts Just a note: I've been seeing this in a lot of tutorials lately. This is not needed unless you have a weird location for the hosts file (ie /home/youraccount/hosts). There are ./configure options that should be used to integrate correctly with the OS filesystem. This fixes a lot of file and folder paths. Details in the squid wiki about each OS type. refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 Add this right here: refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 40% 4320 ##Define your network below #acl mynetwork src 192.168.0.0/24 acl mynetwork src 192.168.106.0/24 # cbinetwork private acl mynetwork src 192.168.107.0/24 # cbinetwork private acl mynetwork src 192.168.110.0/24 # cbinetwork private acl mynetwork src 192.168.120.0/24 # cbinetwork private acl mynetwork src 192.168.121.0/24 # cbinetwork private acl mynetwork src 192.168.130.0/24 # cbinetwork private acl mynetwork src 192.168.150.0/24 # cbinetwork private acl mynetwork src 192.168.151.0/24 # cbinetwork private acl mynetwork src 192.168.160.0/24 # cbinetwork private acl mynetwork src 10.100.101.0/24 # cbinetwork private acl manager proto cache_object acl localhost src 127.0.0.1/32 acl localhost src ::1/128 acl to_localhost dst 127.0.0.0/8 acl to_localhost dst ::1/128 acl purge method PURGE acl CONNECT method CONNECT acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 1025-65535 #unregistered ports acl SSL_ports port 443 563 http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge Um do you actually need PURGE? If not remove it entirely from the config. Including the ACL definition. Simply defining it makes Squid do more work tracking
Re: [squid-users] Missing username on logs when using c-icap
On 23/10/10 11:24, Carlos Xavier wrote: Hi Amos. Based on the information of the configuration o sent before do you think I am missing something or should I report it as a bug. Squid properly passes the user to c-icap but it seans to forget it is one authenticated request and dont set the username on the log. Can't see anything obvious in the config. The proxy-auth headers are hop-by-hop. Which means Squid is expected strip them when contacting any external server. We have a bug open requesting that ICAP be extended with a login= parameter the same as cache_peer. The X-Authenticated-User hack only passes the details for ICAP logging, it does not do a return trip to Squid. I am suspecting that c-icap is re-writing the request and sending a whole new one to Squid to be used instead of the original. But sans the credentials which were not passed over. A workaround, if that is the case, will be for ICAP to use the continue features of ICAP. Where it scans then sends a simple use the original, its fine message back to Squid. I'm not sure right now what the support levels of that are in either c-icap+clamav, your url-rewriter, and Squid-3.1 (likely only fully working in 3.2). Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] Re: cache_dir aufs min-size
On 23/10/10 06:30, Costin Gusa wrote: Hi, I was wondering if someone could shed a light on my previous unanswered email. Thank you. On Tue, Oct 19, 2010 at 22:10, Costin Gusacosti...@gmail.com wrote: Hi, I would like to limit the minimum object size in a cache dir For this I have setup the following line in squid.conf: cache_dir aufs /var/spool/squid 65536 16 256 min-size=4MB IIRC the option=XX parser has not yet been fixed to handled byte sizes like that. The above means minimum 4 bytes. Your earlier min-size=4194304 should have worked. Run more on the small file and see what URL its saved from and what the HTTP headers were. It could be a few things: * one of a few internal files (netdb amd clientdb) which require saving to disk somewhere and pick the first cache_dir. * a large file still arriving from the network. * unknown-length files which are assumed to have an extremely big size until they have arrived. Which has not yet been erased. However looking at /var/spool/squid cache dir, I see smaller objects also: /var/spool/squid# ls -lah 00/00|grep K|head -n3 drwxr-x--- 2 proxy proxy 4.0K 2010-10-19 21:56 . drwxr-x--- 258 proxy proxy 8.0K 2010-10-19 20:31 .. -rw-r- 1 proxy proxy 73K 2010-10-19 20:35 I tried also with min-size=4194304 with no luck Is this an intended behaviour or am I misunderstanding this functionality? [...] ii squid-common2.7.STABLE3-4.1lenny1Internet object cache (WWW proxy c Also, please try an upgrade to the backports.org version in case this is a fixed bug already. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] Youtube upload cpu problem
On 23/10/10 01:50, Marcelo Grassi F. Melgaço wrote: Is there a way to limit upload speed for domain upload.youtube.com in squid? Maybe this help to resolve my problem.. Thanks Not with any of the stable or older releases. Upload speed delay pools has just been accepted for addition to 3.2 today. It should be in the net 3.2 beta release. The patch can be found under Client-side bandwidth limits in squid-dev. I've no idea if it will apply to 3.1 cleanly yet. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
Re: [squid-users] top reports twice memory as much as Total in mgr:mem
On 22/10/10 04:07, Kaiwang Chen wrote: Amos, The cache size is configured around 128GB, as reported by mgr:config: cache_dir aufs /export/squid/cache 131072 32 256 IOEngine=DiskThreads cache_swap_low 90 cache_swap_high 95 cache_mem -1073741824 bytes memory_replacement_policy lru cache_replacement_policy lru memory_pools on memory_pools_limit 5242880 bytes You have probably hit bug http://bugs.squid-cache.org/show_bug.cgi?id=3068 There is a patch in the report waiting feedback. If you can test it and see if it fixes your problem please Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.8 Beta testers wanted for 3.2.0.2
