RE: [squid-users] RE: Forcing squid to cache files
Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done with the override-* and ignore-* is tell Squid that the objects at a each URL never change and not to believe any spftware which states otherwise. When the web server or app producing them may be stating explicitly that they will at a certain timestamp or already have changed. Luckily they only apply to the squid they are set in so other caches doing similar bandwidth reduction outside yours will not be crippled. What you need to do is check the accuracy of the headers being produced by the app and bug its developers to fix any problems you find. For both the server app and the client agent app. Note that 3.1 has almost full HTTP/1.1 support when talking to servers but only HTTP/1.0 features are reliable when talking to the client/visitors. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: Sadly Google is mostly filled with vocal people using 2.5 or jumping to conclusions. Or like yourself with a very narrow focus of HTTP and a specific problem app. refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors
RE: [squid-users] RE: Forcing squid to cache files
Nevermind, if that was the case, I wouldn't see any SWAPOUT lines. Still stumped on this one. Anyone have any tips on how to debug this further? A very small amount of my data is actually being cached. =( -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 10:00 AM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done with the override-* and ignore-* is tell Squid that the objects at a each URL never change and not to believe any spftware which states otherwise. When the web server or app producing them may be stating explicitly that they will at a certain timestamp or already have changed. Luckily they only apply to the squid they are set in so other caches doing similar bandwidth reduction outside yours will not be crippled. What you need to do is check the accuracy of the headers being produced by the app and bug its developers to fix any problems you find. For both the server app and the client agent app. Note that 3.1 has almost full HTTP/1.1 support when talking to servers but only HTTP/1.0 features are reliable when talking to the client/visitors. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: Sadly Google is mostly filled with vocal people using 2.5 or jumping to conclusions. Or like yourself with a very narrow focus of HTTP and a specific problem app. refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore
RE: [squid-users] RE: Forcing squid to cache files
Update: I configured firefox to use my proxy server, and I see squid caching items on normal web browsing. I also see many cache hits in my access.log, so I believe my cache is working correctly. I imagine there's something funky in the http headers we use in our proprietary tool which is causing the data to not be cached correctly. Anyone have any ideas about what that could be? Here's some sample output when running squidclient: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 17:08:04 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=.xen Cache-Control: private Content-Type: xen Content-Length: 64880640 X-Cache: MISS from Via: 1.0 tibp4proxy (squid/3.1.9) Connection: close I have this line in my squid.conf, so I don't think the Cache-Control: private is doing it: refresh_pattern . 7200100%7200 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private Anyone have any ideas? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 10:01 AM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Nevermind, if that was the case, I wouldn't see any SWAPOUT lines. Still stumped on this one. Anyone have any tips on how to debug this further? A very small amount of my data is actually being cached. =( -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 10:00 AM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done with the override-* and ignore-* is tell Squid that the objects at a each URL never change and not to believe any spftware which states otherwise. When the web server or app producing them may be stating explicitly that they will at a certain timestamp or already have changed. Luckily they only apply to the squid they are set in so other caches doing similar bandwidth reduction outside yours will not be crippled. What you need to do is check the accuracy of the headers being produced by the app and bug its developers to fix any problems you find. For both the server app and the client agent app. Note that 3.1 has almost full HTTP/1.1 support when talking to servers but only HTTP/1.0 features are reliable when talking to the client
RE: [squid-users] RE: Forcing squid to cache files
Okay, so I think I figured out why stuff is not saving to my cache correctly. Most of the files I'm transferring are 4 MB, and the default max size for cached files is 4MB. I added the following line to my squid.conf file, and the saving to cache problem seems to be resolved. maximum_object_size 2097152 KB In my opinion, this should be very clearly stated in a GOTCHAS section, as this very simple fix took me a WHILE to figure out ;) So in case anyone else runs into this problem, that's the solution. -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 12:19 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Update: I configured firefox to use my proxy server, and I see squid caching items on normal web browsing. I also see many cache hits in my access.log, so I believe my cache is working correctly. I imagine there's something funky in the http headers we use in our proprietary tool which is causing the data to not be cached correctly. Anyone have any ideas about what that could be? Here's some sample output when running squidclient: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 17:08:04 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=.xen Cache-Control: private Content-Type: xen Content-Length: 64880640 X-Cache: MISS from Via: 1.0 tibp4proxy (squid/3.1.9) Connection: close I have this line in my squid.conf, so I don't think the Cache-Control: private is doing it: refresh_pattern . 7200100%7200 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private Anyone have any ideas? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 10:01 AM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Nevermind, if that was the case, I wouldn't see any SWAPOUT lines. Still stumped on this one. Anyone have any tips on how to debug this further? A very small amount of my data is actually being cached. =( -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 10:00 AM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done
RE: [squid-users] RE: Forcing squid to cache files
So the cache is filling up correctly now, but I still get all TCP_MISS/200 in my access.log. I don't see these issues when caching web pages. Anyone got any ideas? =) -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 2:14 PM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Okay, so I think I figured out why stuff is not saving to my cache correctly. Most of the files I'm transferring are 4 MB, and the default max size for cached files is 4MB. I added the following line to my squid.conf file, and the saving to cache problem seems to be resolved. maximum_object_size 2097152 KB In my opinion, this should be very clearly stated in a GOTCHAS section, as this very simple fix took me a WHILE to figure out ;) So in case anyone else runs into this problem, that's the solution. -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 12:19 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Update: I configured firefox to use my proxy server, and I see squid caching items on normal web browsing. I also see many cache hits in my access.log, so I believe my cache is working correctly. I imagine there's something funky in the http headers we use in our proprietary tool which is causing the data to not be cached correctly. Anyone have any ideas about what that could be? Here's some sample output when running squidclient: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 17:08:04 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=.xen Cache-Control: private Content-Type: xen Content-Length: 64880640 X-Cache: MISS from Via: 1.0 tibp4proxy (squid/3.1.9) Connection: close I have this line in my squid.conf, so I don't think the Cache-Control: private is doing it: refresh_pattern . 7200100%7200 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private Anyone have any ideas? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 10:01 AM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Nevermind, if that was the case, I wouldn't see any SWAPOUT lines. Still stumped on this one. Anyone have any tips on how to debug this further? A very small amount of my data is actually being cached. =( -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 10:00 AM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another
RE: [squid-users] RE: Forcing squid to cache files
Also, it seems that when I use squidclient to get the same file 2 times in a row, the cache DOES behave as expected. First call outputs this: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 19:25:11 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=*.xen Cache-Control: private Content-Type: xen Content-Length: 4718592 X-Cache: MISS from * Via: 1.0 * (squid/3.1.9) Connection: close While the second call outputs this: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 19:25:11 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=*.xen Cache-Control: private Content-Type: xen Content-Length: 4718592 Age: 22 X-Cache: HIT from * Via: 1.0 * (squid/3.1.9) Connection: close Any idea why I would get a hit when calling squidclient from the machine running squid, but when I try to connect from another machine, the files are not served from the cache? The other machine is allowed with an acl. Thanks, -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 2:22 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files So the cache is filling up correctly now, but I still get all TCP_MISS/200 in my access.log. I don't see these issues when caching web pages. Anyone got any ideas? =) -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 2:14 PM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Okay, so I think I figured out why stuff is not saving to my cache correctly. Most of the files I'm transferring are 4 MB, and the default max size for cached files is 4MB. I added the following line to my squid.conf file, and the saving to cache problem seems to be resolved. maximum_object_size 2097152 KB In my opinion, this should be very clearly stated in a GOTCHAS section, as this very simple fix took me a WHILE to figure out ;) So in case anyone else runs into this problem, that's the solution. -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 12:19 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Update: I configured firefox to use my proxy server, and I see squid caching items on normal web browsing. I also see many cache hits in my access.log, so I believe my cache is working correctly. I imagine there's something funky in the http headers we use in our proprietary tool which is causing the data to not be cached correctly. Anyone have any ideas about what that could be? Here's some sample output when running squidclient: HTTP/1.0 200 OK Date: Fri, 10 Dec 2010 17:08:04 GMT Server: Microsoft-IIS/6.0 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Content-Disposition: attachment; filename=.xen Cache-Control: private Content-Type: xen Content-Length: 64880640 X-Cache: MISS from Via: 1.0 tibp4proxy (squid/3.1.9) Connection: close I have this line in my squid.conf, so I don't think the Cache-Control: private is doing it: refresh_pattern . 7200100%7200 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private Anyone have any ideas? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Friday, December 10, 2010 10:01 AM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Nevermind, if that was the case, I wouldn't see any SWAPOUT lines. Still stumped on this one. Anyone have any tips on how to debug this further? A very small amount of my data is actually being cached. =( -Original Message- From: Volker-Yoblick, Adam Sent: Friday, December 10, 2010 10:00 AM To: Volker-Yoblick, Adam; 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files Another related question: I notice that the lastmod and expires values for every line in my store.log is -1. Is squid unable to cache files without lastmod and expires headers? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 9:59 PM To: 'Amos Jeffries'; 'squid-users@squid-cache.org' Subject: RE: [squid-users] RE: Forcing squid to cache files No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid
[squid-users] RE: Forcing squid to cache files
Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:42 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] Forcing squid to cache files Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? Thanks for your time, -Adam
[squid-users] RE: Forcing squid to cache files
I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. Anyone have any ideas why all the files aren't getting cached? Did I miss something in my config file? Thanks for your time, -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:51 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] RE: Forcing squid to cache files Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:42 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] Forcing squid to cache files Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? Thanks for your time, -Adam
[squid-users] RE: Forcing squid to cache files
I noticed that the extensions of the files not being cached are not present in the mime.conf that squid uses. Is it correct to assume that all extensions must be present in the mime types table in order for the file to be cached? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 6:07 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] RE: Forcing squid to cache files I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. Anyone have any ideas why all the files aren't getting cached? Did I miss something in my config file? Thanks for your time, -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:51 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] RE: Forcing squid to cache files Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:42 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] Forcing squid to cache files Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? Thanks for your time, -Adam
[squid-users] RE: Forcing squid to cache files
Hmmm nevermind, that seemed to be incorrect. Anyone have any ideas about why my cache only has 500 MB in it for a 7GB set of files? I can provide snippets of logs if needed. -Original Message- From: Volker-Yoblick, Adam Sent: Thursday, December 09, 2010 6:13 PM To: Volker-Yoblick, Adam; 'squid-users@squid-cache.org' Subject: RE: Forcing squid to cache files I noticed that the extensions of the files not being cached are not present in the mime.conf that squid uses. Is it correct to assume that all extensions must be present in the mime types table in order for the file to be cached? -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 6:07 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] RE: Forcing squid to cache files I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. Anyone have any ideas why all the files aren't getting cached? Did I miss something in my config file? Thanks for your time, -Adam -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:51 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] RE: Forcing squid to cache files Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam [mailto:avol...@ea.com] Sent: Thursday, December 09, 2010 5:42 PM To: 'squid-users@squid-cache.org' Subject: [squid-users] Forcing squid to cache files Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? Thanks for your time, -Adam
Re: [squid-users] RE: Forcing squid to cache files
-Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done with the override-* and ignore-* is tell Squid that the objects at a each URL never change and not to believe any spftware which states otherwise. When the web server or app producing them may be stating explicitly that they will at a certain timestamp or already have changed. Luckily they only apply to the squid they are set in so other caches doing similar bandwidth reduction outside yours will not be crippled. What you need to do is check the accuracy of the headers being produced by the app and bug its developers to fix any problems you find. For both the server app and the client agent app. Note that 3.1 has almost full HTTP/1.1 support when talking to servers but only HTTP/1.0 features are reliable when talking to the client/visitors. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: Sadly Google is mostly filled with vocal people using 2.5 or jumping to conclusions. Or like yourself with a very narrow focus of HTTP and a specific problem app. refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? The big gotcha you have found already is that by ignoring the CC headers that 0 comes into effect immediately. The HTTP headers may permit much longer caching times than the minimum minutes if you only override the CC headers specifically which are problems. ignore-private is particularly dangerous to use. It is sent when the content is destined for exactly one visitor and contains details only they are safe to show. Think banking details or -secret government files level of security. Overriding this will send such files to *all* visitors. no-store is similar with a little bit less danger when the data leaks. Things using this might be shared by simultaneous
RE: [squid-users] RE: Forcing squid to cache files
No, none of the files are 2GB. I know that's a limitation, that's why I mentioned it. =) Anyone else know why the cache might not be populating correctly? -Original Message- From: Amos Jeffries [mailto:squ...@treenet.co.nz] Sent: Thursday, December 09, 2010 9:39 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] RE: Forcing squid to cache files -Original Message- From: Volker-Yoblick, Adam I have another related question: I can see my cache filling up, but I'm sending about 7 gigs through the proxy, and the cache doesn't even have 300 MB in it yet, and the transfer is at 62%. Looking in the store.log, I see a mix of RELEASE and SWAPOUT lines. Also, none of the files are 2 GB. If you are trying to store 2GB individual files, Squid has an accounting bug which screws up the size measures for the cache. see http://bugs.squid-cache.org/show_bug.cgi?id=3068 This has been fixed for 3.1.10, and the bug fix snapshot bundles of 3.1 already contain it. -Original Message- From: Volker-Yoblick, Adam Doh! I feel like a moron. Read up on the refresh_pattern command, and it seems that first 0 on the last line was causing everything to be marked as not fresh right away. I upped that value, and my cache is now filling up. Nothing to see here=) -Original Message- From: Volker-Yoblick, Adam Greetings squid users, I recently installed squid 3.1.9 on an RHEL 5 server, with no options when running ./configure. We have a proprietary tool that sends files from one machine to another over HTTP, and I wanted to have squid always cache the files to help improve transfer times when the tool is used from outside the building. Note that this cache will NEVER be used to serve webpages, so I don't care about violating HTTP protocol. HTTP protocol is not about web pages. HTTP protocol is about reliable transfer and delivery of up-to-date and valid objects. What you have done with the override-* and ignore-* is tell Squid that the objects at a each URL never change and not to believe any spftware which states otherwise. When the web server or app producing them may be stating explicitly that they will at a certain timestamp or already have changed. Luckily they only apply to the squid they are set in so other caches doing similar bandwidth reduction outside yours will not be crippled. What you need to do is check the accuracy of the headers being produced by the app and bug its developers to fix any problems you find. For both the server app and the client agent app. Note that 3.1 has almost full HTTP/1.1 support when talking to servers but only HTTP/1.0 features are reliable when talking to the client/visitors. I was able to set up the acls to allow a source connection from my machine, and to allow a destination connection to another machine. I can tell this is working, because when I start the transfer, I see lots of HTTP GET lines in my access.log. I also see lots of lines in my store.log, but unfortunately, all the lines are RELEASE lines, meaning nothing is being stored in the cache. I verified this by running du -hs on my cache dir, and the size is never going up. I've spent most of the day googling this issue (and looking at the squid FAQ), and it seems most users have the problem where they are not ignoring no-cache commands in the http headers. I tried to get around this in my squid.conf, shown below: Sadly Google is mostly filled with vocal people using 2.5 or jumping to conclusions. Or like yourself with a very narrow focus of HTTP and a specific problem app. refresh_pattern ^ftp: 144020% 10080 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern ^gopher:14400% 1440 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private refresh_pattern . 0 20% 4320 override-expire override-lastmod ignore-reload ignore-no-cache ignore-no-store ignore-must-revalidate ignore-private This doesn't seem to fix the problem, however. I also made sure that the squid user is the owner of my cache_dir, and I made sure that cache_effect_user is set to squid. Running squid -z returns no errors. I'm kinda stumped at this point. Anyone have any suggestions? Maybe a gotcha that I missed, or proper steps to debug this further? The big gotcha you have found already is that by ignoring the CC headers that 0 comes into effect immediately. The HTTP headers may permit much longer caching times than the minimum minutes if you only override the CC headers specifically which are problems. ignore
