Re: [squid-users] Recommend for hardware configurations
On lör, 2008-07-05 at 12:44 -0300, Michel wrote: I am not understanding why you keep suggesting single core as preferred cpu Did I? Not what I can tell. I said Squid uses only one core. even if squid's core is actually not multi-thread capable a faster cpu is better - there are also other things running on a machine so a smp machine ever is a benefit to overall performance Both yes and no. For an application like Squid you will find that nearly all OS:es gets bound to a single core running both networking and the application, leaving the other cores to run various tiny other stuff.. modern OS also should give squid's aufs threading benefits but I am not totally aufs isn't very cpu hungry. It's main purpose is to be able to exploit the parallellism there is in the harddrive hardware. The Squid cache function gets quite seek intensive so there is a huge benefit of being able to have multiple concurrent I/O operations (especially open/create calls...). diskd also isn't very cpu hungry. In fact probably a bit less than aufs. But diskd can not push the drives as far as aufs, and is still plauged some instability issues.. Why I recommend dual core instead of quad core is simply because you get a faster core speed in dual core than quad core for the same price (and often availability as well..) which will directly benefit Squid in high performance. Yes, Squid quite easily gets CPU bound, and is then limited to the core speed of your CPU, and the faster the core speed is the better in that situation. Selecting a slower core speed to fit more cores hurts performance for Squid when the server is mainly for Squid. sure about your design here but at least diskd when running several diskd processes is getting benefits from multicore cpus - and a lot and if you do not believe it set up squid/disk on a 8-core machine and compare with 1|2|4|8 or more diskds to your single-core-cpu-thing and measure it, in fact you do not even measure it, you can see it and smell it ... You are welcome to give numbers proving that for Squid a 4 core system outperforms a 2 core system with the exact same configuration in all other aspects. Don't forget to include price in the matrix.. The most interesting test configurations is - no disk cache - single drive for disk cache - 4 drives for disk cache Until I see any numbers indicating quad core gives a significant increase outperforming what the same price configuration using dual core I will continue propagating that quad core is not beneficial to Squid. Similarly for dual core vs single core, but it's not as clear cut as there is not a big per core performance difference between single and dual core compared to prices.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Recommend for hardware configurations
The cpu doesn't do any IO, it's WAITING for the disk most of the time. If you want fast squid performance, CPU speed/count is irrelevant; get more disks and ram. When I mean more disk, I mean more spindles. eg: 2x 100GB will is better than a 200GB disk. well well, get prepared ... take your cpu out and then you'll see who is waiting forever :) even if IO wait is an issue it is or better WAS one on old giant lock systems where the cpu was waiting until getting the lock on a busy thread because there was only ONE CPU and even on multi-cpu-systems there was only one core a time bound to the kernel to get around this issue good old posix aio_*calls where used in order not to wait for a new lock what I believe is squid's aufs cache_dir model which is still very good and even better on modern smp machines and even with squid's not-smp-optimized code - you really can drain disks to their physical limits - but that is not all SMP (modern) works around the global giant lock, the kernel is not anymore limited to get one core a time SMP sistems are going to work with spin locks (Linux) and sleep locks (freebsd) where the linux way is focusing thread synchronizing which is going to be outperformanced by the sleep lock mechanism. Spin locks certainly still waste cpu while spinning what sleeplocks do not, cpu is free to do other work. This was kind of benefit for Linux last couple of years when freebsd was in deep developing of it's new threading model which is now on top I think, especially in shared memory environments. basicly is it not important if you use one or ten disks, this you should consider later as fine tuning but the threading model works the same, for one or two disks, or for 2 our 32Gigs of memory - so you certainly do NOT get araound your IO-Wait with more memory or more disk when the cpu(s) can not handle it waiting for locks as you say ... So IMO your statement is not so very true anymore, with a modern SMP-OS on modern smp hardware of course. michel Tecnologia Internet Matik http://info.matik.com.br Sistemas Wireless para o Provedor Banda Larga Hospedagem e Email personalizado - e claro, no Brasil.
Re: [squid-users] Recommend for hardware configurations
On lör, 2008-07-05 at 12:44 -0300, Michel wrote: I am not understanding why you keep suggesting single core as preferred cpu Did I? Not what I can tell. I said Squid uses only one core. :) good answer ... but often it does not matter what we say but what is beeing understood, what I meant is that it comes over as if you are suggesting single core computers even if squid's core is actually not multi-thread capable a faster cpu is better - there are also other things running on a machine so a smp machine ever is a benefit to overall performance Both yes and no. For an application like Squid you will find that nearly all OS:es gets bound to a single core running both networking and the application, leaving the other cores to run various tiny other stuff.. nope, not at all. probably on Linux's spin lock model it might be so , but I do not know, on freebsd you can watch the squid process and it's threads, either aufs or diskd related and see that they are handled by all cpus all the time 35867 squid4 -19 3921M 3868M kqread 3 200:28 0.00% squid0 1481 squid4 -19 601M 581M kqread 0 86:03 0.00% squid1 1482 squid4 -19 598M 579M kqread 0 84:49 0.00% squid2 1495 squid -4 -19 8300K 1376K msgrcv 1 20:19 0.00% diskd-daemon 1496 squid -4 -19 8300K 1372K msgrcv 3 20:11 0.00% diskd-daemon 1497 squid -4 -19 8300K 1324K msgrcv 3 5:42 0.00% diskd-daemon 1498 squid -4 -19 8300K 1224K msgrcv 2 5:31 0.00% diskd-daemon 35867 squid4 -19 3921M 3868M kqread 1 200:28 0.00% squid0 1481 squid4 -19 601M 581M kqread 1 86:03 0.00% squid1 1482 squid4 -19 598M 579M kqread 1 84:49 0.00% squid2 1495 squid -4 -19 8300K 1376K msgrcv 0 20:19 0.00% diskd-daemon 1496 squid -4 -19 8300K 1372K msgrcv 0 20:11 0.00% diskd-daemon 1497 squid -4 -19 8300K 1324K msgrcv 2 5:42 0.00% diskd-daemon 1498 squid -4 -19 8300K 1224K msgrcv 2 5:31 0.00% diskd-daemon 35867 squid4 -19 3921M 3868M kqread 1 200:29 0.00% squid0 1481 squid4 -19 601M 581M kqread 2 86:03 0.00% squid1 1482 squid4 -19 598M 579M kqread 3 84:50 0.00% squid2 1495 squid -4 -19 8300K 1376K msgrcv 1 20:19 0.00% diskd-daemon 1496 squid -4 -19 8300K 1372K msgrcv 1 20:11 0.00% diskd-daemon 1497 squid -4 -19 8300K 1324K msgrcv 2 5:42 0.00% diskd-daemon 1498 squid -4 -19 8300K 1224K msgrcv 2 5:31 0.00% diskd-daemon three tops in 3 different seconds, 8'th column show on which cpu it runs, observing threads it still is more fun Why I recommend dual core instead of quad core is simply because you get a faster core speed in dual core than quad core for the same price (and often availability as well..) which will directly benefit Squid in high performance. I understood you recommend single core ... not dual Yes, Squid quite easily gets CPU bound, and is then limited to the core speed of your CPU, and the faster the core speed is the better in that situation. Selecting a slower core speed to fit more cores hurts performance for Squid when the server is mainly for Squid. I am not so sure if the core speed does matter so much as long as there IS CPU left ... then there is CPU left for any other work... You are welcome to give numbers proving that for Squid a 4 core system outperforms a 2 core system with the exact same configuration in all other aspects. Don't forget to include price in the matrix.. The most interesting test configurations is - no disk cache - single drive for disk cache - 4 drives for disk cache Until I see any numbers indicating quad core gives a significant increase outperforming what the same price configuration using dual core I will continue propagating that quad core is not beneficial to Squid. two/three years ago I said next year there are no single cores to buy anymore and everyone is running at least dualcore if not quad and was shot by almost all freebsd 4.x and dragon-fly lovers or should I say by people which didn't saw where the modern threading model was going and were hanging on to the global giant lock because at *THAT* time network and disk performance was still better? then to be honest I do not believe that you ever will be convinced by any test *I* post here :), so do it yourself and get your own conclusions ... the test is easy, get yourself an AM2 MB and a X2 and a X4 and nuke a fixed rate of http requests over a certain time into each CPU and monitor CPU time and disk io (on FREEBSD amd64 7STABLE ) and compare it and I say you show me that X2 is losing and then I get myself a linux box and shut my mouth :) Similarly for dual core vs single core, but it's not as clear cut as there is not a big per core performance difference between single and dual core compared to prices.. as I said soon there are no single cores
Re: [squid-users] Recommend for hardware configurations
* Henrik Nordstrom [EMAIL PROTECTED]: diskd also isn't very cpu hungry. In fact probably a bit less than aufs. But diskd can not push the drives as far as aufs, and is still plauged some instability issues.. So you say aufs is defacto FASTER than diskd? -- Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED] Charite - Universitätsmedizin BerlinTel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-BerlinFax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to [EMAIL PROTECTED]
Re: [squid-users] Recommend for hardware configurations
2008/7/6 Henrik Nordstrom [EMAIL PROTECTED]: Until I see any numbers indicating quad core gives a significant increase outperforming what the same price configuration using dual core I will continue propagating that quad core is not beneficial to Squid. My numbers here say, if your Squid workload is memory-locked, you won't benefit from quad-core. If your Squid workload is CPU-bound, you will benefit from Quad-Core. This applies for intel with their single style memory bus. AMD's are a bit different. Similarly for dual core vs single core, but it's not as clear cut as there is not a big per core performance difference between single and dual core compared to prices.. Well, the benefit comes from being able to run the network TX/RX threads on the other CPU, freeing up the first CPU for running Squid. You'd be surprised how much that will give you. Of course, if Squid wasn't so inefficient when it came to memory copying (ie, it does way too much), it would probably be more CPU rather than memory-bus taxing, and quad-core may start to matter. YMMV. Adrian (Who has single and dual-core AMD/Intel, and Dual dual-core intel xeon in testing, and understands what the hell is going on here..)
[squid-users] cache_peer_domain + POST
Hi all, I have set a cache_peer_domain to use a parent proxy for a specific domain. cache_peer 10.147.113.254 parent218 0 proxy-only name=office cache_peer_domain office .specific.net acl specific dstdomain specific.net never_direct allow specific It will work fine until i try to post a form 1215348582.324285 127.0.0.1 TCP_MISS/200 2310 GET http://host.specific.net/ - FIRST_UP_PARENT/10.147.113.254 text/html 1215348582.435 49 127.0.0.1 TCP_MISS/304 457 GET http://host.specific.net/style/index.css - CD_PARENT_HIT/10.147.113.254 text/css 1215348582.442 53 127.0.0.1 TCP_MISS/304 458 GET http://host.specific.net/webdesign/bandeau-v-musicfinder.gif - CD_PARENT_HIT/10.147.113.254 image/gif 1215348582.448 55 127.0.0.1 TCP_MISS/304 457 GET http://host.specific.net/webdesign/bouton-go.gif - CD_PARENT_HIT/10.147.113.254 image/gif 1215348589.843 24 127.0.0.1 TCP_MISS/403 550 POST http://host.soecific.net/ - DIRECT/xxx.xxx.xxx.xxx text/html Why POST method try DIRECT ? What's wrong? Thanks in advance. Best Regards, Sébastien WENSKE
Re: [squid-users] Need help. Squid won't start
On Sun, Jun 29, 2008 at 3:27 PM, Henrik Nordstrom [EMAIL PROTECTED] wrote: On sön, 2008-06-29 at 10:15 -0500, Nick Lehman wrote: I figured out what was happening. I ended up doing a chown to the squid user on those files. Now I'm having another issue. None of the sites that I have blacklists for under squidguard are being blocked. I see the 4 processes running along side squid, but nothing is being block. Even the expression filters in the squid config are being ignored. Usually it's for the same reasons, permission issues. If SquidGuard detects a proble with it's configuration it enters passthru mode letting all requests pass.. Check your cache.log, at around the time where Squid starts SquidGuard.. And doublecheck that your cache_effective_user has read permission on the SquidGuard configuration data. Regards Henrik Still no luck. I checked everything (cache log doesn't get created) and it's all accessible. the contents of the squidGuard directories are owned by the squid user. I'm at a point now where I get proxy refusing connections I notice that squid starts and then shortly there after stops. Did a -X and it appears to have taken the config. I'm out of ideas.
Re: [squid-users] Need help. Squid won't start
Scratch that, back to square two. Squid and the squidguard processes are running. Proxy is accepting connections, but nothing is being blocked. From what I've read on the url_redirect_program command, squidGuard should take over on all filtering and ignore the squid config. All squid will do is pass the traffic to squidGuard and that is that. So, I'm thinking that the config in squid is correct, but something may be wrong with the squidGuard config. Am I barking up the wrong tree here? -Nick On Sun, Jul 6, 2008 at 9:08 AM, Nick Lehman [EMAIL PROTECTED] wrote: On Sun, Jun 29, 2008 at 3:27 PM, Henrik Nordstrom [EMAIL PROTECTED] wrote: On sön, 2008-06-29 at 10:15 -0500, Nick Lehman wrote: I figured out what was happening. I ended up doing a chown to the squid user on those files. Now I'm having another issue. None of the sites that I have blacklists for under squidguard are being blocked. I see the 4 processes running along side squid, but nothing is being block. Even the expression filters in the squid config are being ignored. Usually it's for the same reasons, permission issues. If SquidGuard detects a proble with it's configuration it enters passthru mode letting all requests pass.. Check your cache.log, at around the time where Squid starts SquidGuard.. And doublecheck that your cache_effective_user has read permission on the SquidGuard configuration data. Regards Henrik Still no luck. I checked everything (cache log doesn't get created) and it's all accessible. the contents of the squidGuard directories are owned by the squid user. I'm at a point now where I get proxy refusing connections I notice that squid starts and then shortly there after stops. Did a -X and it appears to have taken the config. I'm out of ideas.
[squid-users] Combined log showing 0/0 for status/bytes?
Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) Thanks, Tuc
Re: [squid-users] Recommend for hardware configurations
Well, I based my argument from the 10 instances of reverse proxies I'm running. It has 266,268,230 objects and 3.7 TB of space. CPU usage is always around 0.2 according to ganglia. So unless you have some other statistics to prove CPU is that important, I'm stick w/ my argument that disk and RAM is way more important that CPU. mike At 03:41 AM 7/6/2008, Michel wrote: The cpu doesn't do any IO, it's WAITING for the disk most of the time. If you want fast squid performance, CPU speed/count is irrelevant; get more disks and ram. When I mean more disk, I mean more spindles. eg: 2x 100GB will is better than a 200GB disk. well well, get prepared ... take your cpu out and then you'll see who is waiting forever :) even if IO wait is an issue it is or better WAS one on old giant lock systems where the cpu was waiting until getting the lock on a busy thread because there was only ONE CPU and even on multi-cpu-systems there was only one core a time bound to the kernel to get around this issue good old posix aio_*calls where used in order not to wait for a new lock what I believe is squid's aufs cache_dir model which is still very good and even better on modern smp machines and even with squid's not-smp-optimized code - you really can drain disks to their physical limits - but that is not all SMP (modern) works around the global giant lock, the kernel is not anymore limited to get one core a time SMP sistems are going to work with spin locks (Linux) and sleep locks (freebsd) where the linux way is focusing thread synchronizing which is going to be outperformanced by the sleep lock mechanism. Spin locks certainly still waste cpu while spinning what sleeplocks do not, cpu is free to do other work. This was kind of benefit for Linux last couple of years when freebsd was in deep developing of it's new threading model which is now on top I think, especially in shared memory environments. basicly is it not important if you use one or ten disks, this you should consider later as fine tuning but the threading model works the same, for one or two disks, or for 2 our 32Gigs of memory - so you certainly do NOT get araound your IO-Wait with more memory or more disk when the cpu(s) can not handle it waiting for locks as you say ... So IMO your statement is not so very true anymore, with a modern SMP-OS on modern smp hardware of course. michel Tecnologia Internet Matik http://info.matik.com.br Sistemas Wireless para o Provedor Banda Larga Hospedagem e Email personalizado - e claro, no Brasil.
Re: [squid-users] cache_peer_domain + POST
On sön, 2008-07-06 at 15:49 +0200, Sébastien WENSKE wrote: Hi all, I have set a cache_peer_domain to use a parent proxy for a specific domain. cache_peer 10.147.113.254 parent218 0 proxy-only name=office cache_peer_domain office .specific.net acl specific dstdomain specific.net never_direct allow specific It will work fine until i try to post a form You'll need never_direct as well. POST isn't normally cachable so Squid does not bother to use peer caches if it doesn't have to. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Need help. Squid won't start
On sön, 2008-07-06 at 09:33 -0500, Nick Lehman wrote: is that. So, I'm thinking that the config in squid is correct, but something may be wrong with the squidGuard config. Am I barking up the wrong tree here? That's the usual situation when seeing this. And almost always there is some message in cache.log from squidguard telling what is wrong.. Trying to start squidguard manually as your cache_effective_user is also a very good test. Just log in as your cache_effective_user (using sudo or su) and try to run the url_rewrite_program command line from the shell. It should start without any warnings or errors. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Combined log showing 0/0 for status/bytes?
On sön, 2008-07-06 at 13:50 -0400, Tuc at T-B-O-H.NET wrote: Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) On MISS it's usually when the browser aborts the request before headers is known. Not sure what to make out of it on cache hits... but I guess it may be that the request was aborted before the RESPMOD response from the ICAP server is seen.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] cache_peer_domain + POST
Thanks Henrik, I've try to do this, but I don't know how to specify never_direct POST method only for this specific domain, i need to get this parent proxy fot this domain because it's IP filtred and only the IP of my office is allowed. Sebastien. - Original Message - From: Henrik Nordstrom [EMAIL PROTECTED] To: Sébastien WENSKE [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Sunday, July 06, 2008 9:10 PM Subject: Re: [squid-users] cache_peer_domain + POST On sön, 2008-07-06 at 15:49 +0200, Sébastien WENSKE wrote: Hi all, I have set a cache_peer_domain to use a parent proxy for a specific domain. cache_peer 10.147.113.254 parent218 0 proxy-only name=office cache_peer_domain office .specific.net acl specific dstdomain specific.net never_direct allow specific It will work fine until i try to post a form You'll need never_direct as well. POST isn't normally cachable so Squid does not bother to use peer caches if it doesn't have to. Regards Henrik
Re: [squid-users] cache_peer_domain + POST
On sön, 2008-07-06 at 21:57 +0200, Sébastien WENSKE wrote: Thanks Henrik, I've try to do this, but I don't know how to specify never_direct POST method only for this specific domain, i need to get this parent proxy fot this domain because it's IP filtred and only the IP of my office is allowed. Sorry, read your configuration again and I now see that you did try to use never_direct. But a . was missing from your acl.. (but present in cache_peer_domain..) Try this: acl specific dstdomain .specific.net cache_peer 10.147.113.254 parent218 0 proxy-only name=office cache_peer_access office allow specific never_direct allow specific that cache_peer_access line is equivalent to a cache_peer_domain using the same domains, but as you also need an acl matching these domains cache_peer_access is easier as there is less duplication and less risk for unnoticed configuration errors... Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] cache_peer_domain + POST
It works fine !! thank you so much !!! - Original Message - From: Henrik Nordstrom [EMAIL PROTECTED] To: Sébastien WENSKE [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Sunday, July 06, 2008 10:51 PM Subject: Re: [squid-users] cache_peer_domain + POST On sön, 2008-07-06 at 21:57 +0200, Sébastien WENSKE wrote: Thanks Henrik, I've try to do this, but I don't know how to specify never_direct POST method only for this specific domain, i need to get this parent proxy fot this domain because it's IP filtred and only the IP of my office is allowed. Sorry, read your configuration again and I now see that you did try to use never_direct. But a . was missing from your acl.. (but present in cache_peer_domain..) Try this: acl specific dstdomain .specific.net cache_peer 10.147.113.254 parent218 0 proxy-only name=office cache_peer_access office allow specific never_direct allow specific that cache_peer_access line is equivalent to a cache_peer_domain using the same domains, but as you also need an acl matching these domains cache_peer_access is easier as there is less duplication and less risk for unnoticed configuration errors... Regards Henrik
RE: [squid-users] httpReadReply: Request not yet fully sent POSThttp://xxx/yyy.php
On tor, 2008-07-03 at 15:00 +0100, Joe Tiedeman wrote: It seems to be that IIS is sending the 401 response before squid the client have finished sending the initial request to it, after sniffing the traffic with wireshark on the client, squid is forwarding the 401 response before the client has finished posting the data. The interesting things is what happens after the 401 response. Do Squid close the connection before the client sent all of the request, or is the connection kept open allowing the client to continue sending the request? What about the connection squid-webserver? The microsoft schemes NTLM / Negotiate and Kerberos is a bit at odds with how HTTP authentication works, which causes some quite odd corner cases.. How things are supposed to work in the HTTP way is that the connection is kept open and request data being read, but the client when seeing the 401 should immediately abort the transfer (by closing the connection) and try again with correct credentials. This can not be done in the connection oriented auth schemes and the client must instead transmit the whole request, even when it's known it is now going into the bitbucket.. may not be such a big deal when on a LAN/Intranet, but if over a WAN it can be very annoying.. Regards Henrik signature.asc Description: This is a digitally signed message part
RE: [squid-users] httpReadReply: Request not yet fully sentPOSThttp://xxx/yyy.php
Hi Henrik, Just today I came across this bug report submitted in January http://www.squid-cache.org/bugs/show_bug.cgi?id=2176 which appears to match my situation almost exactly. I'm at home at the moment so can't check exactly, but it appears that squid keeps the connection open and the client continues sending the request. Only sometimes (but noticeably often) does the connection appear to be ungracefully closed and the browser errors saying it was reset. I haven't packet sniffed squid - webserver yet, but I can do and post the results if that would help. It only appears to reset the connection on the larger posts, the guy who originally posted the bug report said it's posts over 8kb, I've yet to verify that, but certainly smaller sized posts work without issue. Yeah I'm aware that the MS auth schemes weren't exactly designed to work in harmony with HTTP authentication! Unfortunately needs must!! If you can offer any advice, or let me know what else I can to do help diagnose the issue, I'd be most grateful. Cheers Joe -Original Message- From: Henrik Nordstrom [mailto:[EMAIL PROTECTED] Sent: Sunday 6 July 2008 22:54 To: Joe Tiedeman Cc: squid-users@squid-cache.org Subject: RE: [squid-users] httpReadReply: Request not yet fully sentPOSThttp://xxx/yyy.php; On tor, 2008-07-03 at 15:00 +0100, Joe Tiedeman wrote: It seems to be that IIS is sending the 401 response before squid the client have finished sending the initial request to it, after sniffing the traffic with wireshark on the client, squid is forwarding the 401 response before the client has finished posting the data. The interesting things is what happens after the 401 response. Do Squid close the connection before the client sent all of the request, or is the connection kept open allowing the client to continue sending the request? What about the connection squid-webserver? The microsoft schemes NTLM / Negotiate and Kerberos is a bit at odds with how HTTP authentication works, which causes some quite odd corner cases.. How things are supposed to work in the HTTP way is that the connection is kept open and request data being read, but the client when seeing the 401 should immediately abort the transfer (by closing the connection) and try again with correct credentials. This can not be done in the connection oriented auth schemes and the client must instead transmit the whole request, even when it's known it is now going into the bitbucket.. may not be such a big deal when on a LAN/Intranet, but if over a WAN it can be very annoying.. Regards Henrik _ Higher Education Statistics Agency Ltd (HESA) is a company limited by guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ. Registered No. 2766993. The members are Universities UK and GuildHE. Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799. HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA, registered in England at the same address. Registered No. 3109219. _ This outgoing email was virus scanned for HESA by MessageLabs. _
Re: [squid-users] Combined log showing 0/0 for status/bytes?
Tuc at T-B-O-H.NET wrote: Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) Thanks, Tuc Hey Tuc, what were the requirements that still hold you to 2.6? Just speed performance or something else? Amos -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7
Re: [squid-users] Combined log showing 0/0 for status/bytes?
On s=C3=B6n, 2008-07-06 at 13:50 -0400, Tuc at T-B-O-H.NET wrote: Hi, =20 I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : =20 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.fa= cebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/= 1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE= 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIREC= T =20 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.fac= ebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebo= ok.com/home.php? Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET = CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT =20 =20 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) On MISS it's usually when the browser aborts the request before headers is known. Ok.. Don't know if the user is doing that or not... Not sure what to make out of it on cache hits... but I guess it may be that the request was aborted before the RESPMOD response from the ICAP server is seen.. ICAP server? Even though its compiled into the system, I don't have any ICAP enabled : #Default: # icap_enable off valhalla# grep ^icap squid.conf valhalla# (I'm one of those ENABLE EVERYTHING JUST INCASE type of people. ;) ) Thanks, Tuc
Re: [squid-users] how safe is server_http11?
FWIW, I've tested it, and have been using it in production on a fair number of boxes for a little while; so far so good. Like H says, the main thing is lacking Expect/Continue support. Cheers, On 04/07/2008, at 6:55 AM, Chris Woodfield wrote: So we're looking to upgrade from 2.6 to 2.7, primarily to get the HTTP/1.1 header support. I realize that the full 1.1 spec is not completely implemented, but are there any real Danger, Will Robinson! implications? Specifically, is there any functionality or access to content that would be actively broken because squid is advertising HTTP/1.1 but doesn't have the spec completely implemented? Thanks, -C -- Mark Nottingham [EMAIL PROTECTED]
Re: [squid-users] Combined log showing 0/0 for status/bytes?
Tuc at T-B-O-H.NET wrote: Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) Thanks, Tuc Hey Tuc, what were the requirements that still hold you to 2.6? Just speed performance or something else? Well, I'd say squid FreeBSD Ports, but I should be at 2.6.21 then. I don't see 2.7 there, so thats my reason for 2.7. squid30 is at 3.0.7, so for that I plead insanity. Actually, this site has had some many other problems in the last few months, coming up to speed on versions has been taken off the list of TODO. Though, if your saying I get speed/performance with 2.6 over the others... Then I guess I'm not too worried. Thanks, Tuc
Re: [squid-users] Combined log showing 0/0 for status/bytes?
FWIW, I've seen this on hits as well with 2.[6,7]... I assumed it was either a very immediate abort, or the log tag being set incorrectly (which AIUI happens sometimes, as tcp_hit is the default, no?). Cheers, On 07/07/2008, at 5:15 AM, Henrik Nordstrom wrote: On sön, 2008-07-06 at 13:50 -0400, Tuc at T-B-O-H.NET wrote: Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) On MISS it's usually when the browser aborts the request before headers is known. Not sure what to make out of it on cache hits... but I guess it may be that the request was aborted before the RESPMOD response from the ICAP server is seen.. Regards Henrik -- Mark Nottingham [EMAIL PROTECTED]
[squid-users] LRU Statistics
I have the following cache stats: Memory hits as % of hit requests: 5min: 51.7% Disk hits as % of hit requests: 5min: 31.8% Storage Mem capacity: 100.0% used, 0.0% free i.e., cache miss in mem = 100 - 51.7 = 48.3% overal hit % = 51.7 + 31.8 = 83.5% 1. Since memory is now 100% used, how do I know if there is a cache miss in mem 48.3%, how many % of them will trigger a LRU in memory cache? 2. Similarly, if my disk is 100% used, how do I know the LRU statistics Thanks.
[squid-users] Squid and ziproxy
Hi, Is it possible to use squid with ziproxy(http://ziproxy.sourceforge.net/)? If so, what is the usual configuration? If not, how can I implement a ziproxy-like HTML/JS/CSS optimization in Squid? Is there a pluggable module interface for this purpose? Regards, Kwang Yul Seo
[squid-users] Request Header contains NULL characters :is that sloved
This is regarding the posting: Request header contains NULL characters. http://www.mail-archive.com/squid-users@squid-cache.org/msg16754.html I see back in 2004, Mozilla browser gives this error. But are there any improvements to this. I use Mozilla and get these errors for some website [even yahoo.com]. IS there any workaround. Thanks Geetha DISCLAIMER: This email (including any attachments) is intended for the sole use of the intended recipient/s and may contain material that is CONFIDENTIAL AND PRIVATE COMPANY INFORMATION. Any review or reliance by others or copying or distribution or forwarding of any or all of the contents in this message is STRICTLY PROHIBITED. If you are not the intended recipient, please contact the sender by email and delete all copies; your cooperation in this regard is appreciated.
Re: [squid-users] Combined log showing 0/0 for status/bytes?
Tuc at T-B-O-H.NET wrote: Tuc at T-B-O-H.NET wrote: Hi, I'm running squid/2.6.STABLE20+ICAP via WCCP2. I seem to be getting more and more instances of : 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://photos-c.ak.facebook.com/photos-ak-sctm/v43/18/2433486906/app_2_2433486906_3650.gif HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_HIT:DIRECT 192.168.3.249 - - [05/Jul/2008:19:08:44 -0400] GET http://profile.ak.facebook.com/v230/847/104/q748084879_3980.jpg HTTP/1.1 0 0 http://www.facebook.com/home.php?; Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727) TCP_MISS:DIRECT 0 status and 0 bytes sent. Is this something that just happens ? (193 out of 2909 hits yesterday) Thanks, Tuc Hey Tuc, what were the requirements that still hold you to 2.6? Just speed performance or something else? Well, I'd say squid FreeBSD Ports, but I should be at 2.6.21 then. I don't see 2.7 there, so thats my reason for 2.7. squid30 is at 3.0.7, so for that I plead insanity. Actually, this site has had some many other problems in the last few months, coming up to speed on versions has been taken off the list of TODO. Though, if your saying I get speed/performance with 2.6 over the others... Then I guess I'm not too worried. You do, 2.6/2.7 are approx 25% faster than 3.0. I thought you may have been one of those who are stuck with 2.6+ICAP because of other missing features in 3.0. I'm on track to push those ahead now if any others are brought to my attention. I see from the other posts that you may not actually be using the ICAP feature. - If you are not actually needing it, I'd advise dropping the patch from your 2.x squid. Some of your problems may be attributable to the incomplete nature of the 2.6 ICAP patch. - If ICAP is important to you, you should have 3.0 on your planning list for testing, as the stability and support for ICAP in 3.x is very much better. Amos -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7
Re: [squid-users] Request Header contains NULL characters :is that sloved
Geetha_Priya wrote: This is regarding the posting: Request header contains NULL characters. http://www.mail-archive.com/squid-users@squid-cache.org/msg16754.html I see back in 2004, Mozilla browser gives this error. But are there any improvements to this. I use Mozilla and get these errors for some website [even yahoo.com]. IS there any workaround. Thanks Geetha Not within squid. There is nothing Squid can do if it received bad request data. You will have to check that your 'Mozilla' (whichever of the many hundreds of browsers that call themelves that) is up to date and contact it's developers about why its sending NULL characters in HTTP requests. Amos -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7
Re: [squid-users] LRU Statistics
Roy M. wrote: I have the following cache stats: Memory hits as % of hit requests: 5min: 51.7% Disk hits as % of hit requests: 5min: 31.8% Storage Mem capacity: 100.0% used, 0.0% free i.e., cache miss in mem = 100 - 51.7 = 48.3% overal hit % = 51.7 + 31.8 = 83.5% Note the text as % of hit requests So... cache hit in mem = 51.7% cache hit in disk = 31.8% cache hit in other (peers? refresh?) = 100 - 51.7 - 31.8 = 16.5% miss information completely unknown. 1. Since memory is now 100% used, how do I know if there is a cache miss in mem 48.3%, how many % of them will trigger a LRU in memory cache? Good question. 2. Similarly, if my disk is 100% used, how do I know the LRU statistics Disk won't get 100% used unless you make your cache too big for the drive, or screw up cache gargbage collection with the cleanup settings. Amos -- Please use Squid 2.7.STABLE3 or 3.0.STABLE7
Re: [squid-users] adding a parameter to a URL / Problem in the url_redirect program
Dear Friends,
Thanks a lot for all the support so far. Little by little i have some
solutions. But my redirect programs behave very unusual way. Its a working
script. How would i say , when i pass argument via comand line , result was
as i expected. but when its in squid , it doesnt act how it supposed to
work.
All the web requests comes into the redirect program , its adding VALUE-X
value into the script. But here i am doing grep for ? mark in to
requested url , if its matched doing another function squid itself. it was
success.
When a web request receives as follows ,
http://10.48.15.1:8080/main/main_view?id=7
that adding of the value wont happen.why is that ? why its not perform in
squid redirector program ? doi have to do any thing special in squid or in
the perl script ?
Following is my script.
#!/usr/bin/perl
# no buffered output, auto flush
use strict;
use warnings;
my ($temp, $array, @array, $param_1, $param_2, $param_3, $new_uri);
$|=1;
$temp = ;
while (STDIN){
[EMAIL PROTECTED] = split(/ /);
($param_1, $param_2, $param_3) = split(/ /);
#if (!($array[1] =~ m#VALUE-X#)) {
if (!($param_2 =~ m#VALUE-X#)) {
$temp = $param_2;
if ($param_2 =~ m#\?#) {
$temp .= VALUE-X=652224848;
}else {
$temp .= ?VALUE-X=652224848;
}
$new_uri = ($param_1 . . $temp . . $param_3);
s#$param_2#$temp#;
#print $new_uri;
print;
}else {
print;
}
}
Thank you
Shaine.
Marcus Kool wrote:
Shaine,
Because you use the 302: prefix the URL that you pass back from the
redirector
to Squid is sent back to the browser and because of the 302 the browser
sends a new request to Squid and the new URL is the URL that the
redirector sent
in the first place. This URL is passed by Squid to the redirector and now
you
have an endless loop.
solution:
- do not use the 302: prefix, or
- modify your perl code so that it checks for orchard=66677722 in which
case it must not
rewrite the URL.
-Marcus
Shaine wrote:
Dear friedns,
I am really fed up with url -redirecting program which i have used for
redirect specific url comes and validate by the url -rewriting program .
It
wasnt gave me any errors at initial steps.But now when i try to access
google or any url , its repating ( url * n ) ,
eg:-
:http://www.yahoo.com/302:http://www.yahoo.com/302:http://www.yahoo.com/302:http://www.yahoo.com/302:http://www.yahoo.com/302:http://www.yahoo.com/302:http://www.yahoo.com
following is my perl code ( url-redirecting program )
#!/usr/bin/perl
# no buffered output, auto flush
$|=1;
$temp = ;
while (STDIN){
@array = split(/ /);
if (!(@array[1] =~ m#orchard#)) {
$temp = 302: . @array[1];
if (@array[1] =~ m/\?/) {
$temp .= orchard=66677722;
}else {
$temp .= ?orchard=66677722;
}
[EMAIL PROTECTED];
print;
}else {
print;
}
}
Can somebody help me to solve my issue ? please help me. actually why it
is
happening like that .
Many thanks
Shaine.
Sylvain Viart-2 wrote:
Hi Shaine,
Shaine a écrit :
I have a big problem with adding a parameter to a URL which passes via
squid
. For that i am going to use url_rewrite program. I had a big time with
squid url rewriting, but no success.
Could you please tell me , to get in to my point what are the minimum
requirement to be satisfied ?
I haven't tested to rewrite the querystring part of the url, but it's
available on the redirector (rewrite_program)
Here's sample input for the rewrite_program
0
http://www.somedomain.com/thumb/100/3/b/2/7/3b279a6eab3d0a983d9tre.somedomain.com/messenger/messPing.php
12.34.56.78/- - POST -
0
http://subdom.somedomain.com/thumb/55/3/c/3/6/3c36046ed06c78b2b65627f660be6220.jpg
12.34.56.78/- - GET -
0
http://www.somedomain.com/thumb/100/3/6/8/4/3684949288972604fafdb167ffc214d5.jpg
12.34.56.78/- - GET -
0
http://www.somedomain.com/thumb/100/7/a/4/1/7a4113fd5fba8ec93fa6bf82a6c993be.jpg
12.34.56.78/- - GET -
0
http://www..somedomain.com/thumb/100/4/3/d/f/43df2ca304f508557294d3a835a6fd29.jpg
12.34.56.78/- - GET -
The digit in the first position is only present when
url_rewrite_concurrency is used, see
The thread : url_rewrite_concurrency singlethreaded redirector
performance?
http://www.mail-archive.com/squid-users@squid-cache.org/msg49897.html
url_rewrite_program
url_rewrite_children
url_rewrite_concurrency
url_rewrite_host_header on|off
url_rewrite_access allow|deny acl ...
I use :
url_rewrite_program /etc/squid/redirector.pl
url_rewrite_children 100
url_rewrite_concurrency 50
url_rewrite_host_header off
which means :
100 process spawned (busy proxy)
url_rewrite_concurrency 50, means squid can pass up to 50 URL to the
program using a counter
url_rewrite_host_header off, means that redirector rewrites the URL, but
squid keep the original URL, useful in
