Re: [squid-users] Squid log formats - 2.5-2.6?
File a bug so it's not forgotten. Easy to review i code. On tis, 2008-06-17 at 13:15 +1000, Mark Nottingham wrote: That reminds me; when using logformat, I've seen some counters show up as '-' when the value is 0. I can try to reproduce if more info is needed... On 17/06/2008, at 8:26 AM, Henrik Nordstrom wrote: On mån, 2008-06-16 at 11:21 -0400, Mike Diggins wrote: Has something changed in the access log format between Squid 2.5Stable14 and 2.6Stable20? I'm just upgrading and noticed my webalizer can no longer parse the access.log file. It complains about the date which I believe is the same on both (seconds since the Epoch). Error: Skipping record (bad date): [31/dec/1969:19:00:00 -] [68] In squid 2.6, I've picked the default squid format (logformat): logformat squid %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %un %Sh/ %A %mt Should work. Are you sure you told webalizer to parse a Squid access log in Squid native format, and not a common log format? Also try without the logformat directive. The squid format is built-in, and may differ sligtly if you redefine it with a logformat directive.. Regards Henrik -- Mark Nottingham [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: [squid-users] No auth, only log?
On tis, 2008-06-17 at 06:48 +0700, docdiz wrote: Squid team had made somthing better than they knew :-) No but many applications tunnel their stuff over http, and often on port 80. RTSP (Real) is a good example of this, using tho HTTP requests to establis a bidirectoinal opaqueue (and completely uncacheabe) tunnel over HTTP proxies. Other applications are abusing the CONNECT method to do the same thing. meant to be used for SSL, but is in reality being used a lot more for other traffic such as FTP, IRC, Peer-To-Peer and god knows what.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Cnnecting from squid with http 1.1 to an origin server is enable?
On tis, 2008-06-17 at 13:51 +0900, S.KOBAYASHI wrote: However let me ask you that does SQUID 3-HEAD support to connect and send to the origin server with http 1.1? Not yet. Only 2.7 has experimental support for this. Regards Henrik signature.asc Description: This is a digitally signed message part
[squid-users] Re: Help with sarg usage
In any case - the report seems to cover the whole period of the log. Even though the report is generates every 30 minutes - it appears to cover the whole squid log period. YES Is there any way to restrict the report to a short period (say 1 hour) of within the coverage of the squid log. I still do not know. I think it is good to send another mail with the subject of restrict access log to a short period (say 1 hour) Then, squid developers might be able to answer you. go ahead to bring this to an end Happy Squiding. -- Thank you Indunil Jayasooriya
Re: [squid-users] High CPU usage and degraded service time after 2 weeks of activity
Tony, On Mon, Jun 16, 2008 at 7:31 PM, Anthony Tonns [EMAIL PROTECTED] wrote: Did you ever find a resolution to this issue? I'm running a very similar config and running into very similar problems - only on more servers using more memory and the RHEL squid package on CentOS 5 x86_64. Same symptoms - no paging going on, only using 5.5G of the 8G of ram. It will run fine for a few days. But then squid will totally consume 1 of the 4 cores in the system (two dual-core AMD Opteron(tm) Processor 2212) but after restart only 10-20% of one core. The only significant difference other than sizing is that I have memory_replacement_policy set at lru instead of heap GDSF. No. I let the broken Squid run for a few days, waiting for someone helping us diagnosing the problem but as I didn't have any answer, we restarted Squid as the service was really degraded. It solved the problem and we haven't reproduced it for now. FYI, we don't have a lot of regexp rules (a few refresh patterns and around 20 user-agents acls). I tried to oprofile the production Squid when we got the problem but didn't succeed in it. If someone has a good oprofile tutorial, I'm more than interested as I didn't find anything useful yet. Be sure I'll keep the list informed if I have some news about the problem. -- Guillaume
Re: [squid-users] Squid log formats - 2.5-2.6?
JPP wrote: On Mon, 16 Jun 2008 16:46:16 -0600, JPP wrote On Tue, 17 Jun 2008 00:26:41 +0200, Henrik Nordstrom wrote On mån, 2008-06-16 at 11:21 -0400, Mike Diggins wrote: Has something changed in the access log format between Squid 2.5Stable14 and 2.6Stable20? I'm just upgrading and noticed my webalizer can no longer parse the access.log file. It complains about the date which I believe is the same on both (seconds since the Epoch). Error: Skipping record (bad date): [31/dec/1969:19:00:00 -] [68] In squid 2.6, I've picked the default squid format (logformat): logformat squid %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %un %Sh/%A %mt Should work. Are you sure you told webalizer to parse a Squid access log in Squid native format, and not a common log format? Also try without the logformat directive. The squid format is built-in, and may differ sligtly if you redefine it with a logformat directive.. Regards Henrik Greetings all! To quickly chime in, I had a very similar problem and it turned out to be my squid.conf file and the: logformat squid %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %un %Sh/%A %mt line. Believe it or not, there was a SPACE between 'squid' and '%ts' and the result was a SPACE in the logs before the date started showing. Made me nuts too until I started digging and on a lark, saw it, removed the space and it started working again. I did not report it mainly because I had not seen any others mentioning it, and figured it was some bonehead thing I did! Could still be, of course. Hopefully this may be your problem also. And hope it helps if it is! Regards, JPP Sorry - let me be clearer - there were TWO spaces, the one space that is there is necessary, mine had an extra one. JPP Thanks you very much for noticing that. I've committed a patch to Squid-3 and Squid-2 to correct the config docs. It should percolate out to any future releases. Amos -- Please use Squid 2.7.STABLE1 or 3.0.STABLE6
Re: [squid-users] No auth, only log?
On tis, 2008-06-17 at 11:06 +0200, Falk wrote: I meant that others didn't need to be auth'ed, only cached in squid. But they are all HTTP when being used via the HTTP proxy. Maybe (but only maybe) can you match on user-agent however, basing the auth requirement on the application used. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Is it possible to have squid as do Proxy and OWA/RPCoHTTPS accelerator?
Alan Lehman wrote: My current config (which works for OWA, but not RPCoHTTP): extension_methods RPC_IN_DATA RPC_OUT_DATA https_port public_ip_for_owa:443 cert=/usr/share/ssl/owa/combined.crt key=/usr/share/ssl/owa/owa.key defaultsite=owa.tld.com https_port public_ip_for_rpc:443 cert=/usr/share/ssl/rpc/combined.crt key=/usr/share/ssl/rpc/rpc.key defaultsite=rpc.tld.com cache_peer ip_of_exchange parent 80 0 no-query originserver front-end-https=auto login=PASS You need a second entry for port 443 on the exchange server to handle the RPC requests. This is where the name= parameter becomes very important and needs to be unique for each entry and used in the cache_peer_access lines below. Thanks for the reply and clarification on the OWA config. My intention was to pass the RPC to Exchange unencrypted, on port 80. Previously, either you or Henrik had suggested eliminating one of the cache_peer lines, since both OWA and RPC would be going to port 80 on Eggchange. Oh yes, I'd overlooked that earlier mail (wasn't mine). I'm skeptical that Exchange will allow RPC over non-secure ports. Or if its listening there, that it would tell clients the port-80 on Squid. The squid config you had _should_ work for Squid under that setup. I think form teh non-working state you reported there is something missing at the Exchange end to get both on port-80. ... if you can get it going over port-80 great. I'd like to know how for the wiki. Amos -- Please use Squid 2.7.STABLE2 or 3.0.STABLE6
[squid-users] temp countermeasure against swap.state corruption
hi the swap.state corruption is a real problem. Since I have no time for learning the squid sources and find out what it is I wrote a workaround which seams to protect from this to happen. the swap.state corruption is appearing after squid receives the first requests while rebuilding the swap.state. In latest versions the -F flag does not help anymore, some weeks ago ( 2.6-stable19) it still was a valid workaround. So what my startup script does is injecting a firewall rule blocking any incoming tcp:8080, reading the log, detecting when swap.state is ready and then on single instances remove the initial firewall rule, or in multi instance scenario start the process which receive the client requests only when the swap.dirs are ready. So if someone is interested ask me in pvt or if I do not step on someones tail here I can post it to the list. michel ... Tecnologia Internet Matik http://info.matik.com.br Sistemas Wireless para o Provedor Banda Larga Hospedagem e Email personalizado - e claro, no Brasil.
[squid-users] error
Hi all, Can some one explain me the bellow message? temporary disabling (Bad Gateway) digest from 10.200.2.16 10.200.2.16 is my parent proxy (trend micro apps) Regards, Remy
[squid-users] Re[squid-users] verse proxy to Sharepoint
We have a working infrastructure using Windows 2003, AD Sharepoint for Project Web Access. In order to allow branch office access, we wanted to put in place a reverse proxy solution and looked at Squid. After a lot of reading, it became clear the Squid 2.6 or above was the best option in order to get working NTLM authentication. So We've installed a Fedora Core 9 box with Squid 3.0, attached it to the domain and set up all the kerberos, ldap authentication etc. However, it's not quite behaving correctly. Testing kerberos, ldap etc. seems all OK and the ntlm helpers are running OK. Connect to the proxy with IE of Firefox and the request for a password is presented but regardless of what is entered authentication. Obviously I need to provide more information but can you guide me as to where and what I need to provide? Thanks -- View this message in context: http://www.nabble.com/Reverse-proxy-to-Sharepoint-tp17909397p17909397.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Low performance in tranfers duration
Hi !!! I installed squid 3 stable on a Debian box, with 1Gb of RAM, 160 Gb of disk and AMD Optreon Dual Core, in transparent mode. The squid box have only one nework card and public IP. I have a Mikrotik router-box, and in this machine I have configured to send al http request (port 80) to port 3128 of my squid box. Everything seams work find, but sometimes I have very low performance in tranfers duration. I note in the cache.log that I have a lot of warnings messages with forwarding loops detected, is the only error or warning I found. I don't put any routing rules in my squid box. It's ok? May be I have a configuration problem. Any Help? Thanks a lot !!! Ramiro
[squid-users] regex wildcard for refresh_pattern
Is the regex wildcard for refresh_pattern a '.*' or just a '.'. I want to apply a pattern match to some specific jar files that fit a pattern of 'name.jar-serialnumber.zip' Would the correct pattern match then be 'refresh_pattern -i name.jar.*' ?
[squid-users] other questions re: pattern-matching
I assume I can specify multiple options for a specific regex pattern...such as ignore-no-cache and ignore-refresh? And that it is space delimited, or how is it delimited? Also, does the regex match against any part of the URL, or do I have to specify the whole URL? I had assumed the regex patches any part of the URL...but my regex is matching. Nick
Re: [squid-users] iptables syntax
On 16.06.08 18:11, Ken W. wrote: Thanks. I have two 1000M cards. does not matter. Does it support all web applications like videos, webIM etc? only HTTP applications. Don't redirect anything but port 80 communication there. And even then you can get complaints when someone runs non-HTTP application on port 80... 2008/6/16 Indunil Jayasooriya [EMAIL PROTECTED]: I will run Squid on Linux OS, with transparent mode. Should I use iptables to do the http intercept? what's the iptables syntax? please help, thank you. How many network card does this squid box have? in squid.conf, Pls add below line http_port 3128 transparent This is the iptables rule #Redirecting traffic destined to port 80 to port 3128 iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 80 -j REDIRECT --to-port 3128 for more, pls visit below URL http://wiki.squid-cache.org/SquidFaq/InterceptionProxy Happy Squiding -- Thank you Indunil Jayasooriya -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. - Have you got anything without Spam in it? - Well, there's Spam egg sausage and Spam, that's not got much Spam in it.
RE: [squid-users] Apple Computers jam my NTLM Helpers.
Hello Henrik. So you believe that the problem is not related to Squid but related with the apple computer ? The thing that I don't understand is that inside a Apple computer, it's a Linux operating system. Linux is able to talk NTLM language. If it's related with Squid, is it only a refresh problem of Squid or the Helper is really use by Squid ? The helpers are jammed, but the Mac computer is able to surf on the internet without problem. I did a test with an Apple computer and when I open Safari, for exemple, to surf on the internet, I get 4-5 helper that are jammed, but even if these helpers are jammed, my authentication is done and I'm able to surf on the internet. In my Access.log, I see the username of the personne with a TCP_HIT, MISS or ... but no Denied. No error message in my Cache.log but get error when my Squid crash because all process are busy. It's the reason why on my side, I think that the problem is related with ntlm_auth from Samba that is not able to release it after the Apple computer did his authentication. There is a new update of Samba-common available to install. I will probably do it soon, but there is nothing in the log indicating a problem with the ntlm_auth module and it's not the first time that I update my Squid server and have a samba-common to install and the problem is still there. If someone else has an idea, solution, give me your solution or test. Thanks. Jonathan Subject: RE: [squid-users] Apple Computers jam my NTLM Helpers. From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] CC: [EMAIL PROTECTED]; squid-users@squid-cache.org Date: Fri, 6 Jun 2008 23:37:41 +0200 On tor, 2008-06-05 at 20:10 -0400, Jonathan Chretien wrote: It's very strange. I really don't know if it's a Mac problem or if it's a problem with the Helper that has difficulty to talk with Mac Computers. Shoule be easy to see with a wireshark capture of the traffic. Each new connection starting an NTLM handshake reserves a helper until the authentication completes or the connection is closed. My guess on what happens is that the client opens a connection, sends the initial negotiate blob, and gets the challenge from the helper and then just sits there doing nothing with the connection, when it's expected to send an authentication blob (final NTLM packet) Regards Henrik _
[squid-users] Squid problem:. Some addresses work OK - but most hang indefinitely.
Hi I installed Squid 2.6 on Centos 5.1 X86_64 system about a week ago - and it worked fine for the first few days. I have set all clients to use the Squid Proxy for all external (non private 192.168.0.0/24) ip addresses. The only squid config settings I changed from default were ACL changes to allow proxy access to everyone on the local network. I now have the following situation on this client: 1) I can browse local addresses fine (as they are direct) 2) I can browse a few non local addresses fine. I can refresh my ISPs usage data OK for example, and it is clearly refreshing the live data via squid. 3) If I browse most arbitrary web addresses - the firefox tab hangs indefinitely with the little circular animation on the tab. 4) If I revert to direct access (Non proxy) - everything works fine. 5) I have deleted the entire cache - and maybe that helped for a bit - but the problem returned very soon after. 6) I have checked CPU and memory usage on the centos machine - and everything looks fine - almost nothing happening. 7) I did make some router changes to try to prevent direct access from clients - but I have since reverted these changes because the router did not behave as expected. It is now back to the starting point - but the problem persists. 8) I have recently installed sarg, Calamaris and Webalizer - but I doubt these could be responsible for the problem. Can anyone suggest what might be going on here, and if so - how to fix it? If not - can anyone advise diagnostic steps? Regards richard.
[squid-users] Squid on steroids
I've been given a directive to build a squid farm on steroids. Load balanced, multiple servers, etc. I've been googling around and found some documentation but does anyone have any direct experience with this? Any suggestions? Thank you in advance.
Re: [squid-users] Squid log formats - 2.5-2.6?
On Mon, 16 Jun 2008, JPP wrote: On Mon, 16 Jun 2008 16:46:16 -0600, JPP wrote On Tue, 17 Jun 2008 00:26:41 +0200, Henrik Nordstrom wrote On mån, 2008-06-16 at 11:21 -0400, Mike Diggins wrote: Has something changed in the access log format between Squid 2.5Stable14 and 2.6Stable20? I'm just upgrading and noticed my webalizer can no longer parse the access.log file. It complains about the date which I believe is the same on both (seconds since the Epoch). Error: Skipping record (bad date): [31/dec/1969:19:00:00 -] [68] In squid 2.6, I've picked the default squid format (logformat): logformat squid %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %un %Sh/%A %mt Should work. Are you sure you told webalizer to parse a Squid access log in Squid native format, and not a common log format? Also try without the logformat directive. The squid format is built-in, and may differ sligtly if you redefine it with a logformat directive.. Regards Henrik Greetings all! To quickly chime in, I had a very similar problem and it turned out to be my squid.conf file and the: logformat squid %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %un %Sh/%A %mt line. Believe it or not, there was a SPACE between 'squid' and '%ts' and the result was a SPACE in the logs before the date started showing. Made me nuts too until I started digging and on a lark, saw it, removed the space and it started working again. I did not report it mainly because I had not seen any others mentioning it, and figured it was some bonehead thing I did! Could still be, of course. Hopefully this may be your problem also. And hope it helps if it is! Regards, JPP Sorry - let me be clearer - there were TWO spaces, the one space that is there is necessary, mine had an extra one. Doh, that was it, two spaces instead of one. The two spaces are in the default squid.conf file so perhaps that's something that could be fixed. Funny, I was comparing my old access log and new log file, and never noticed the leading space! Thanks for your help! -Mike
[squid-users] Log Format
Hello all. Simple question. Is there a way to purge the access log to get only the URL that the user requested when he clicked on a link or when he typed the address in the address bar ? Squid log a lot of stuff in the Access.log but when I need to give an Audit to my HR department, I'm able to purge it by taking only the text/ mime type (text/html, text/asp) but I still get to much information. If I access hotmail.com, this will also give me in the log something like a.rad.live.com, b.rad.live.com, login.live.com. Is there a way to only get hotmail.com and not all accessed web page ? If I go in the History of Internet Explorer, for exemple, I only get the accessed web site, hotmail.com. i do not get the other redirected web page. I need to be able to give a Audit log to my HR department like the History of Internet Explorer. I did some search on the internet but didn't find anything interesting. Thanks. Jonathan _
Re: [squid-users] Squid on steroids
What's your workload? E.g., is it going to be used as a proxy farm for dialup users? Broadband? If so, how many? Or, is it for an accelerator, and if so, how much content is there? Cheers, On 18/06/2008, at 5:07 AM, [EMAIL PROTECTED] wrote: I've been given a directive to build a squid farm on steroids. Load balanced, multiple servers, etc. I've been googling around and found some documentation but does anyone have any direct experience with this? Any suggestions? Thank you in advance. -- Mark Nottingham [EMAIL PROTECTED]
Re: [squid-users] Squid on steroids
More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you. -- Original message -- From: Mark Nottingham [EMAIL PROTECTED] What's your workload? E.g., is it going to be used as a proxy farm for dialup users? Broadband? If so, how many? Or, is it for an accelerator, and if so, how much content is there? Cheers, On 18/06/2008, at 5:07 AM, [EMAIL PROTECTED] wrote: I've been given a directive to build a squid farm on steroids. Load balanced, multiple servers, etc. I've been googling around and found some documentation but does anyone have any direct experience with this? Any suggestions? Thank you in advance. -- Mark Nottingham [EMAIL PROTECTED] More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you.
Re: [squid-users] Squid on steroids
If you're not caching at all and using reasonably modern hardware (e.g., dual core, ~3Ghz), you should be able to get somewhere between 2,000 and 4,000 requests a second out of a single squid process, depending on the average response size. YMMV, of course, and that doesn't count the overhead of the filtering, etc. By 50,000 users, do you mean total (i.e., you have 50,000 customers), or 50,000 a day, or 50,000 concurrently, or...? Figuring out how much capacity you need is an inexact science, of course, but it's usually best to over-provision. The hard part is going to be directing requests to the proxies, and handling failure well. I haven't done ISP proxy deployments in a long time, so I'll leave it to others to give you advice on that part. I'm assuming you'll want it to be transparent (e.g., use WCCP)? On 18/06/2008, at 9:05 AM, [EMAIL PROTECTED] wrote: More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you. -- Original message -- From: Mark Nottingham [EMAIL PROTECTED] What's your workload? E.g., is it going to be used as a proxy farm for dialup users? Broadband? If so, how many? Or, is it for an accelerator, and if so, how much content is there? Cheers, On 18/06/2008, at 5:07 AM, [EMAIL PROTECTED] wrote: I've been given a directive to build a squid farm on steroids. Load balanced, multiple servers, etc. I've been googling around and found some documentation but does anyone have any direct experience with this? Any suggestions? Thank you in advance. -- Mark Nottingham [EMAIL PROTECTED] More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you. -- Mark Nottingham [EMAIL PROTECTED]
RE: [squid-users] Squid on steroids
The hard part is going to be directing requests to the proxies, and handling failure well. I haven't done ISP proxy deployments in a long time, so I'll leave it to others to give you advice on that part. I'm assuming you'll want it to be transparent (e.g., use WCCP)? If transparent, WCCPv2 has cache failure detection and load balancing. I imagine it would be the easiest/cheapest method if your routers support it.
Re: [squid-users] Squid on steroids
50,000 customers total. We're looking at LVS using keepalived for load balancing at the front-end. I'm most interested in the squid back-end setup. Should we look at something linux based clustering? Or should we be looking at some internal squid process? Could we run multiple squid processes on multi-processor servers? Is that wise? Or should we look at more servers rather than multiple squid processes on one server? How much memory for each server? I've seen some talk here about oprofile. We plan on checking that out to see where things are. I have no problem in setting up a test environment and reporting the results back here. I'd just like to start out on the right track. I do believe in giving back to the community with information we'll learn through this process. Thank you for your comments and thoughts thus far. -- Original message -- From: Mark Nottingham [EMAIL PROTECTED] If you're not caching at all and using reasonably modern hardware (e.g., dual core, ~3Ghz), you should be able to get somewhere between 2,000 and 4,000 requests a second out of a single squid process, depending on the average response size. YMMV, of course, and that doesn't count the overhead of the filtering, etc. By 50,000 users, do you mean total (i.e., you have 50,000 customers), or 50,000 a day, or 50,000 concurrently, or...? Figuring out how much capacity you need is an inexact science, of course, but it's usually best to over-provision. The hard part is going to be directing requests to the proxies, and handling failure well. I haven't done ISP proxy deployments in a long time, so I'll leave it to others to give you advice on that part. I'm assuming you'll want it to be transparent (e.g., use WCCP)? On 18/06/2008, at 9:05 AM, [EMAIL PROTECTED] wrote: More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you. -- Original message -- From: Mark Nottingham [EMAIL PROTECTED] What's your workload? E.g., is it going to be used as a proxy farm for dialup users? Broadband? If so, how many? Or, is it for an accelerator, and if so, how much content is there? Cheers, On 18/06/2008, at 5:07 AM, [EMAIL PROTECTED] wrote: I've been given a directive to build a squid farm on steroids. Load balanced, multiple servers, etc. I've been googling around and found some documentation but does anyone have any direct experience with this? Any suggestions? Thank you in advance. -- Mark Nottingham [EMAIL PROTECTED] More broadband connections than anything else. Possibly as many as 50,000 users. No accelerator, maybe not even caching. Mostly to filter downloads, record websites, etc. maybe with something like urldb or Dansguardian. Do you have ideas??? Thank you. -- Mark Nottingham [EMAIL PROTECTED]
Re: [squid-users] Log Format
Hello all. Simple question. Is there a way to purge the access log to get only the URL that the user requested when he clicked on a link or when he typed the address in the address bar ? That is exactly what the access.log contains, a record of the pages and files requested by the user. Squid log a lot of stuff in the Access.log but when I need to give an Audit to my HR department, I'm able to purge it by taking only the text/ mime type (text/html, text/asp) but I still get to much information. If I access hotmail.com, this will also give me in the log something like a.rad.live.com, b.rad.live.com, login.live.com. That is because the users machine requested objects from all those sources. There is no easy way to identify what resources a page uses without parsing and processing each page yourself or writing a very fancy log parser that tries to detect which requests are the first in a page. Is there a way to only get hotmail.com and not all accessed web page ? If I go in the History of Internet Explorer, for exemple, I only get the accessed web site, hotmail.com. i do not get the other redirected web page. I need to be able to give a Audit log to my HR department like the History of Internet Explorer. I did some search on the internet but didn't find anything interesting. What the users see in their browser is very often completely different to how the web works. The cleanest solution you will get to this whole problem is to accept all the strange domains listed in access.log and give HR a report listing them and popularity/day etc. Amos
Re: [squid-users] Squid problem:. Some addresses work OK - but most hang indefinitely.
Hi I installed Squid 2.6 on Centos 5.1 X86_64 system about a week ago - and it worked fine for the first few days. I have set all clients to use the Squid Proxy for all external (non private 192.168.0.0/24) ip addresses. The only squid config settings I changed from default were ACL changes to allow proxy access to everyone on the local network. I now have the following situation on this client: 1) I can browse local addresses fine (as they are direct) 2) I can browse a few non local addresses fine. I can refresh my ISPs usage data OK for example, and it is clearly refreshing the live data via squid. 3) If I browse most arbitrary web addresses - the firefox tab hangs indefinitely with the little circular animation on the tab. 4) If I revert to direct access (Non proxy) - everything works fine. 5) I have deleted the entire cache - and maybe that helped for a bit - but the problem returned very soon after. 6) I have checked CPU and memory usage on the centos machine - and everything looks fine - almost nothing happening. 7) I did make some router changes to try to prevent direct access from clients - but I have since reverted these changes because the router did not behave as expected. It is now back to the starting point - but the problem persists. 8) I have recently installed sarg, Calamaris and Webalizer - but I doubt these could be responsible for the problem. Can anyone suggest what might be going on here, and if so - how to fix it? If not - can anyone advise diagnostic steps? It sounds like you are hitting one of the interception catch-22s. Only you don't mention interception. Do you have any FW entries specifically for the proxy box? What exactly do your ACL and access lines look like now? Amos
Re: [squid-users] regex wildcard for refresh_pattern
Is the regex wildcard for refresh_pattern a '.*' or just a '.'. . and * have the same meaning in Squid as in any other regex system. We use the OS regex libraries to interpret them. '.' means any single character, * means none-or-more repeating. I want to apply a pattern match to some specific jar files that fit a pattern of 'name.jar-serialnumber.zip' Would the correct pattern match then be 'refresh_pattern -i name.jar.*' ? Just to be sure it only matches them, I'd use: -i name\.jar-[^\.]+\.zip Amos
Re: [squid-users] Low performance in tranfers duration
Hi !!! I installed squid 3 stable on a Debian box, with 1Gb of RAM, 160 Gb of disk and AMD Optreon Dual Core, in transparent mode. The squid box have only one nework card and public IP. I have a Mikrotik router-box, and in this machine I have configured to send al http request (port 80) to port 3128 of my squid box. Everything seams work find, but sometimes I have very low performance in tranfers duration. I note in the cache.log that I have a lot of warnings messages with forwarding loops detected, is the only error or warning I found. I don't put any routing rules in my squid box. It's ok? May be I have a configuration problem. Definately a configuration problem. Squid is trying ask itself for new data whenever that loop is mentioned. You need to make absolutely sure that the router is never sending port-80 requests from the squid IP back to the Squid box. Amos
Re: [squid-users] other questions re: pattern-matching
I assume I can specify multiple options for a specific regex pattern...such as ignore-no-cache and ignore-refresh? And that it is space delimited, or how is it delimited? Whitespace delimited. The pattern needs to be one unit. Though is skips my mind right now how you would indicate whitespace in the pattern. (Maybe [:space:]) Also, does the regex match against any part of the URL, or do I have to specify the whole URL? The pattern matches against any part of the entire URL. If you use regex start/end anchors they are obeyed. Otherwise it may match mid-way. I had assumed the regex patches any part of the URL...but my regex is matching. I assume 'patches' was a typo. The URL is not altered by regex in Squid. Amos
Re: [squid-users] Squid problem:. Some addresses work OK - but most hang indefinitely.
Amos Jeffries wrote: Hi I installed Squid 2.6 on Centos 5.1 X86_64 system about a week ago - and it worked fine for the first few days. I have set all clients to use the Squid Proxy for all external (non private 192.168.0.0/24) ip addresses. The only squid config settings I changed from default were ACL changes to allow proxy access to everyone on the local network. I now have the following situation on this client: 1) I can browse local addresses fine (as they are direct) 2) I can browse a few non local addresses fine. I can refresh my ISPs usage data OK for example, and it is clearly refreshing the live data via squid. 3) If I browse most arbitrary web addresses - the firefox tab hangs indefinitely with the little circular animation on the tab. 4) If I revert to direct access (Non proxy) - everything works fine. 5) I have deleted the entire cache - and maybe that helped for a bit - but the problem returned very soon after. 6) I have checked CPU and memory usage on the centos machine - and everything looks fine - almost nothing happening. 7) I did make some router changes to try to prevent direct access from clients - but I have since reverted these changes because the router did not behave as expected. It is now back to the starting point - but the problem persists. 8) I have recently installed sarg, Calamaris and Webalizer - but I doubt these could be responsible for the problem. Can anyone suggest what might be going on here, and if so - how to fix it? If not - can anyone advise diagnostic steps? It sounds like you are hitting one of the interception catch-22s. Only you don't mention interception. Do you have any FW entries specifically for the proxy box? What exactly do your ACL and access lines look like now? Thanks Amos. Interestingly - whatever the problem was seems to time out after several hours. All is working OK this morning. I don't intentionally have any interceptions. I did try to set up firewall rules for the proxy box - but my firewall/router is a Netgear DG834G - and there seems to be something wrong with its outgoing rules implementation. Specifically - I set up the following rules - in order. always allow any port outgoing from proxy IP. always disallow any port outgoing from all IPs. When I set this up - I had very erratic behaviour. Some web pages came up slowly - and some not at all. There were also problems with fetchmail (running on the same box as squid) downlaoding mail. I attributed this to a problem in the router. When I removed these rules - things reverted to normal - but then a bit later - I had this apparent proxy problem. Given the fact that the problem appears to time out after several hours - I am wondering if there is a DNS issue. I have seen some references to SQUID caching DNS info - but I don't know much about it. If there was a temporary DNS problem at some time - would squid (or something else) cache the DNS miss - and continue returning the miss after the problem was resolved? While the problem was occurring - I did test the DNS server (bind) running on the SQUID box - and it was able to resolve the addresses which were failing via squid. If there is a DNS problem - I don't think it is in the bind server. Thanks again for your advice. Richard. Amos
Re: [squid-users] Squid problem:. Some addresses work OK - but most hang indefinitely.
Amos Jeffries wrote: Hi I installed Squid 2.6 on Centos 5.1 X86_64 system about a week ago - and it worked fine for the first few days. I have set all clients to use the Squid Proxy for all external (non private 192.168.0.0/24) ip addresses. The only squid config settings I changed from default were ACL changes to allow proxy access to everyone on the local network. I now have the following situation on this client: 1) I can browse local addresses fine (as they are direct) 2) I can browse a few non local addresses fine. I can refresh my ISPs usage data OK for example, and it is clearly refreshing the live data via squid. 3) If I browse most arbitrary web addresses - the firefox tab hangs indefinitely with the little circular animation on the tab. 4) If I revert to direct access (Non proxy) - everything works fine. 5) I have deleted the entire cache - and maybe that helped for a bit - but the problem returned very soon after. 6) I have checked CPU and memory usage on the centos machine - and everything looks fine - almost nothing happening. 7) I did make some router changes to try to prevent direct access from clients - but I have since reverted these changes because the router did not behave as expected. It is now back to the starting point - but the problem persists. 8) I have recently installed sarg, Calamaris and Webalizer - but I doubt these could be responsible for the problem. Can anyone suggest what might be going on here, and if so - how to fix it? If not - can anyone advise diagnostic steps? It sounds like you are hitting one of the interception catch-22s. Only you don't mention interception. Do you have any FW entries specifically for the proxy box? What exactly do your ACL and access lines look like now? Thanks Amos. Interestingly - whatever the problem was seems to time out after several hours. All is working OK this morning. I don't intentionally have any interceptions. I did try to set up firewall rules for the proxy box - but my firewall/router is a Netgear DG834G - and there seems to be something wrong with its outgoing rules implementation. Specifically - I set up the following rules - in order. always allow any port outgoing from proxy IP. always disallow any port outgoing from all IPs. Um, I think the problem there is that these rules should only refer to destination port 80 traffic, not 'any port outgoing'. When I set this up - I had very erratic behaviour. Some web pages came up slowly - and some not at all. There were also problems with fetchmail (running on the same box as squid) downlaoding mail. I attributed this to a problem in the router. When I removed these rules - things reverted to normal - but then a bit later - I had this apparent proxy problem. Okay, my comment above should help with those side-effects. But your rules as given would be unrelated to the slow proxy problem. Given the fact that the problem appears to time out after several hours - I am wondering if there is a DNS issue. I have seen some references to SQUID caching DNS info - but I don't know much about it. If there was a temporary DNS problem at some time - would squid (or something else) cache the DNS miss - and continue returning the miss after the problem was resolved? While the problem was occurring - I did test the DNS server (bind) running on the SQUID box - and it was able to resolve the addresses which were failing via squid. If there is a DNS problem - I don't think it is in the bind server. Squid runs a very simple DNS client (A,PTR,SOA records only). It will cache both positive and negative DNS results according to the public DNS SOA records given it by your bind. If your bind was working normally, I would not expect a problem in Squid. Sounds like probably a user request taking up more than it should have for a few hours. (Maybe windows updates range-requests from several clients hitting crunch point?) Amos
Re: [squid-users] Where are the ircache.net cgi for creating graphs?
--- On Fri, 6/13/08, Henrik Nordstrom [EMAIL PROTECTED] wrote: From: Henrik Nordstrom [EMAIL PROTECTED] Subject: Re: [squid-users] Where are the ircache.net cgi for creating graphs? To: [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Date: Friday, June 13, 2008, 3:54 PM On fre, 2008-06-13 at 14:53 -0700, Richard Hubbell wrote: Hello squid world, I was looking for the scripts that create the graphs on ircache.net, I found everything but the cgi scripts. Does anyone know where to get them? Or maybe there's another package that's preferred to make use of RRD for Squid? I have a small script at http://www.henriknordstrom.net/code/ There is also Squid packages to most of the system monitoring tools munin cacti monit Ganglia nagios Zenoss Any other SNMP capable monitoring tools is also able to graph Squid without too much effort Zabbix The above is just a small collection, there is plenty of these tools around.. My experience after making the script mentioned above is that you are most likely better off using a real monitoring package than something special tailored just for Squid. In general I agree. But in this case I wanted something quick and just for squid. And I didn't want to use cgi, just want something that runs out of cron. I'm just trying to understand Squid and what it's doing. Have looked at mod_*cache and now looking at Squid. My main goal is to conserve bandwidth. Thank you Henrik for your help. Regards Henrik
[squid-users] Squid supports all the cache directives described in RFC2616 and SNMPv1 agent?
Hi Squid world, First of all sorry if I am asking someting obvious, I am a really squid newbie. Now I am using squid-2.6.STABLE19 to set up a web proxy, I want to know whether squid-2.6.STABLE19 supports all the cache directives described in the section 13 and 14.9 in IETF RFC2616, or some of them, and works as a SNMPv1 agent? Is there any links or documentations about this? Many thanks! Richard Zou
[squid-users] Async request queue growing uncontrollably!
Hi,all I got a lot of messages when I porting squid2.5 to squid2.6 with coss,who know why? Thanks! === 2008/06/18 11:45:14| squidaio_queue_request: Async request queue growing uncontrollably! 2008/06/18 11:45:14| squidaio_queue_request: Syncing pending I/O operations.. (blocking) 2008/06/18 11:45:14| squidaio_queue_request: Synced 2008/06/18 11:45:33| squidaio_queue_request: WARNING - Disk I/O overloading 2008/06/18 11:45:34| squidaio_queue_request: Async request queue growing uncontrollably! 2008/06/18 11:45:34| squidaio_queue_request: Syncing pending I/O operations.. (blocking) 2008/06/18 11:45:34| squidaio_queue_request: Synced 2008/06/18 11:45:53| squidaio_queue_request: WARNING - Disk I/O overloading 2008/06/18 11:46:05| squidaio_queue_request: Async request queue growing uncontrollably! 2008/06/18 11:46:05| squidaio_queue_request: Syncing pending I/O operations.. (blocking) 2008/06/18 11:46:05| squidaio_queue_request: Synced
[squid-users] custom logformats and squid-2.5.STABLE14-1.4E
I looked around but seem to not have much luck finding or figuring out exactly what logging features work in this version of Squid. Is there a version matrix somewhere listing versions and the features they support? Or maybe I'm just looking in the wrong places? Is it just me or is google becoming less and less useful? With all this SEO stuff it seems like irrelevant stuff percolates up to the top of google results.
