Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi Solomon, Solomon Asare wrote: Hi, --- Manoj_Rajkarnikar [EMAIL PROTECTED] wrote: Great job solomon. Many of us have been trying to achieve similar with youtube and google vids. this will help a great deal. how big of a cachedir do you keep for youtube vids. should be quite a big to be able to cache the vids in large enough quantity to get a decent hit. I'm gonna try to achieve what you've describe here for my next project. Thanks for the job well done. Manoj Currently, my cache size is 70 GB. I have just filled almost 25 GB. I will upgrade to either 200 or 500 GB when I deploy. I really appreciate your initiative and great work regarding caching youtube objects. Everybody will benefit from your work. Are you intending to run a single cache of 500 GB in size or a couple of proxy caches amounting to 500 GB in size? It's because running a 500 GB cache in a single machine is going to cause you problems along the way. Such a large cache will also be a little more difficult to maintain compared to a 10 or 20 GB cache. Despite the low cost of hard drives these days, I would prefer to spread the disk storage over a couple of squid caches instead of a single cache. They can also be configured for load-balancing later. If for some reasons, your cache gets corrupted, it might take a very long time to fix it and I am sure that Squid's median response might will get higher. Anyway, it's just my suggestion. Thanking you... Solomon. -- With best regards and good wishes, Yours sincerely, Tek Bahadur Limbu System Administrator (TAG/TDG Group) Jwl Systems Department Worldlink Communications Pvt. Ltd. Jawalakhel, Nepal http://www.wlink.com.np http://teklimbu.wordpress.com
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi, --- Tek Bahadur Limbu [EMAIL PROTECTED] wrote: Are you intending to run a single cache of 500 GB in size or a couple of proxy caches amounting to 500 GB in size? It's because running a 500 GB cache in a single machine is going to cause you problems along the way. Such a large cache will also be a little more difficult to maintain compared to a 10 or 20 GB cache. Despite the low cost of hard drives these days, I would prefer to spread the disk storage over a couple of squid caches instead of a single cache. They can also be configured for load-balancing later. If for some reasons, your cache gets corrupted, it might take a very long time to fix it and I am sure that Squid's median response might will get higher. Anyway, it's just my suggestion. Now that you have mentioned it I will do some reading on the subject to determine what the optimal disk size should be. May be bigger is not better. Thanks. Solomon.
Re: [squid-users] Caching Expired Objects - One Small Step Forward
On Mon, Oct 08, 2007, Solomon Asare wrote: If for some reasons, your cache gets corrupted, it might take a very long time to fix it and I am sure that Squid's median response might will get higher. Anyway, it's just my suggestion. Now that you have mentioned it I will do some reading on the subject to determine what the optimal disk size should be. May be bigger is not better. Ah, things get a bit weird when you start factoring in huge objects into Squid's cache store. The old logic doesn't hold for large files. One youtube video takes precisely the same amount of memory to index than a small image. It still takes one inode up in the UFS-like storage system, and rebuilding from that store is a factor of the number of files, not of how big those files are. (Squid only reads the header from each file to build the in-memory index, it doesn't read the entire file.) That said, you're currently caching the videos in the apache proxy and not in the Squid proxy. Adrian
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi, --- Adrian Chadd [EMAIL PROTECTED] wrote: That said, you're currently caching the videos in the apache proxy and not in the Squid proxy. Adrian Thanks for the info on large file sizes and memory requirements. I store the objects in squid, not in apache. The apache is a non-caching proxy. Essentially, it rewrites the headers to clear the no-store, expiration, etc., limitations. Thats all. Thanks, solomon.
Re: [squid-users] Caching Expired Objects - One Small Step Forward
On Mon, Oct 08, 2007, Solomon Asare wrote: Thanks for the info on large file sizes and memory requirements. I store the objects in squid, not in apache. The apache is a non-caching proxy. Essentially, it rewrites the headers to clear the no-store, expiration, etc., limitations. Thats all. Aha! Cool. I have to admit, I hadn't looked very deeply at it. Thats weird though. I'm able to cache the resultant videos with: refresh_pattern -i \.flv$ 10080 90% 99 reload-into-ims acl youtube dstdomain .youtube.com cache allow youtube Which worked fine when I last worked on it a couple months ago. So unless they've gone and changed how they serve content again..! Ah. You're not doing rewriting inside Apache or Squid - you're building up jesred rules based on the proxy logs. I see. Cool. It seems they have gone and changed how they serve videos. Argh! I'll have to look some more into it later on. Squid could probably be patched to do what you're using Apache for.. Adrian
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi, --- Adrian Chadd [EMAIL PROTECTED] wrote: I'll have to look some more into it later on. Squid could probably be patched to do what you're using Apache for.. Adrian That will be great, but what I don't understand is why they don't want to make such a static object cacheable. Microsoft, Google, and those whose traffic take up most of the Internet's capacity should make their content cacheable so as to help all of us. At some periods windowsupdate.com takes up about 10% of total traffic with virtually no significant HITs. Thanks, solomon.
Re: [squid-users] Caching Expired Objects - One Small Step Forward
On Mon, Oct 08, 2007, Solomon Asare wrote: That will be great, but what I don't understand is why they don't want to make such a static object cacheable. Microsoft, Google, and those whose traffic take up most of the Internet's capacity should make their content cacheable so as to help all of us. At some periods windowsupdate.com takes up about 10% of total traffic with virtually no significant HITs. Because the market overseas (North America, Europe) is about how many bits can you push and assuming your peer is going to deal with shuffling them back to their customers. Adrian
Re: [squid-users] Caching Expired Objects - One Small Step Forward - WARNING
Hi All, pls should anyone want to try the scripts I posted, do not use /tmp. I irretrievably trashed my 300+ rules which have been built over a week. I now have to crawl up all over again. I wonder why I didn't see this coming. You may use /var/log/squid/. Thanks, solomon.
Re: [squid-users] Caching Expired Objects - One Small Step Forward - WARNING
Ah.. a Gotcha !! Noted. Thanks Manoj On Mon, 8 Oct 2007, Solomon Asare wrote: Hi All, pls should anyone want to try the scripts I posted, do not use /tmp. I irretrievably trashed my 300+ rules which have been built over a week. I now have to crawl up all over again. I wonder why I didn't see this coming. You may use /var/log/squid/. Thanks, solomon. --
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi All, Hi All, a small success: I have set a an apache2.2 server as a proxy without caching as a parent proxy to my squid. I pass my youtube video clips through this parent. The apache is on the same machine as the squid. Using the mod_header module in apache, I cleaned out all the worrisome headers in the the youtube clips and now squid caches them and produces hits. The only outstanding issue is youtube serves videos from random servers. I will therefore have to implement a redirection script so that if a video clip has already been cached, I will pick that one. I hope to work on that over the weekend. I sometimes get about 10% traffic to youtube so if I can get a descent HIT rate it will be significant to the overall performance of my squid. Thanks for the hand-holding, Henrik. Regards, solomon. I promised an update: Following my using an apache proxy as a parent to my squid for youtube videos, I have setup jesred to rewrite my url's for videos already in my cache. HIT rate has gradually moved from about less than 3% to close to 30%. I hope it will continue to increase with time as I get more videos in my cache. I have a script in crontab searching through store.log to update my jesred regex rules. Things To Do: Maintain my jesred rules with a script that will expunge regex rules for expired objects. Thanks, Now ready to deploy my squid for the whole network within the next few days. Windowsupdate still a headache though. Not found much that have been helpful yet. solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi All,
the long skelatel howto:
THE ENVIROMENT.
Ubuntu, Squid 2.6, Apache 2.2, a small ISP with about
11% www.youtube.com traffic, 2Mbps on submarine fibre
to the US cost about US$5,000 - US$7,000 over here -
Accra, Ghana.
THE PROBLEM
Some Youtube clips are not cacheable, even with the
overides provided for in squid2.6. The objects too are
already expired so if you are able to cache them (such
as using Squid3), you are still unable to get HITs.
Moreover, youtube serves the same video from several
servers with different urls, so when it is cached and
not expired, you still may not get a HIT.
PROPOSED SOLUTION, BRIEF:
I use a non-cacheable Apache proxy on the same machine
as the squid, as a parent to the squid for youtube
videos. The apache proxy then sanitises the youtube
clips so that squid can cache them. Using the video ID
as a key, I then use jesred to rewrite the urls for
youtube video objects that are already in my squid
cache so that I will have a HIT.
1 SQUID
i. Modify squid.conf to Cache Youtube Clips.
acl utubevids urlpath_regex get_video\?video
cache allow utubevids
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
ii. Modify squid.conf to handle the size of video
clips.
maximum_object_size 128 MB
cache_dir ufs /var/spool/squid 7 16 256
iii. Modify squid.conf to redirect youtube videos for
url_rewriting.
url_rewrite_program /usr/lib/squid/jesred
acl utuberedir urlpath_regex get\_video\?video\_id\=.*
url_rewrite_access allow utuberedir
iv. Modify squid.conf to use a parent proxy for
youtube videos.
cache_peer 192.168.0.20 parent8000 3130 no-query
default no-netdb-exchange no-digest
acl utubevidsdirect urlpath_regex
get\_video\?video\_id\=.*
always_direct allow !utubevidsdirect
v. Modify refresh_pattern in squid.conf for youtube
video clips.
refresh_pattern -i
get\_video\?video\_id\=.*youtube\.com 10080 990%
99 reload-into-ims ignore-no-cache override-expire
ignore-private
refresh_pattern -i
youtube\.com\/get\_video\?video\_id\=.* 10080 990%
99 reload-into-ims ignore-no-cache override-expire
ignore-private
You could merge the two into one by ignoring the
youtube.com. I am not sure if other sites don't use
the same syntax so I kept it there.
2 APACHE
i. Activate the following modules:
cd /etc/apache2/mods-enabled
ln -s ../mods-available/expires.load
ln -s ../mods-available/headers.load
ln -s ../mods-available/proxy.conf
ln -s ../mods-available/proxy_http.load
ln -s ../mods-available/proxy.load
ii. Configure Apache proxy
(/etc/apache2/mods-enabled/proxy.conf)
IfModule mod_proxy.c
ProxyRequests On
Proxy *
AddDefaultCharset off
Order deny,allow
Deny from all
Allow from 192.168.0
Allow from 127.0.0
Allow from 127.0.1
/Proxy
ProxyVia On
Header unset Cache-Control
Header unset Expires
Header unset Pragma
/IfModule
iii Configure Apache Proxy Port in
/etc/apache2/ports.conf
Listen 8000
3 CONFIGURE THE JESRED REDIRECTOR (url_rewriter).
i. Configure /etc/jesred.conf
allow = /etc/jesred.acl
rules = /etc/jesred.youtube.rules
ii. Configure /etc/jesred.acl
0.0.0.0/0
iii. Configure /etc/jesred.youtube.rules
touch /etc/jesred.youtube.rules
A typical line in your /etc/jesred.youtube.rules
produced by the script below will look like:
regex get\_video\?video\_id\=yDC9iJyTUmc.*
http://74.125.10.23/get_video?video_id=yDC9iJyTUmcorigin=ash-v57.ash.youtube.com
4 SCRIPT TO UPDATE /etc/jesred.youtube.rules kept in
/usr/local/bin/buildrules
#!/bin/bash
cat /etc/jesred.youtube.rules |grep -v ^# |grep -v ^$
|grep 'regex get\\_video\\?video\\_id\\='
/tmp/jesred/jesred.rules.2.tmp
cat /etc/jesred.youtube.rules |grep -v 'regex
get\\_video\\?video\\_id\\='
/tmp/jesred/jesred.rules.1.tmp
cat /var/log/squid/store.log |grep youtube.com |grep
flv |grep SWAPOUT |awk '{split($13,idarray,=);
split(idarray[2],idarr,); print (regex
get\_video\?video\_id\=idarr[1].*, $13);}'
/tmp/jesred/jesred.rules.2.tmp
cat /tmp/jesred/jesred.rules.2.tmp |sort -u
/tmp/jesred/jesred.rules
cp /etc/jesred.youtube.rules
/tmp/jesred/jesred.youtube.rules.old
mv /tmp/jesred/jesred.rules.1.tmp
/etc/jesred.youtube.rules
cat /tmp/jesred/jesred.rules
/etc/jesred.youtube.rules
rm /tmp/jesred/jesred.rules
rm /tmp/jesred/jesred.rules.2.tmp
squid -k rotate
5. ENTRY IN CRONTAB with crontab -e to update my
redirector rules on the 13th minute of every hour.
13 * * * * /usr/local/bin/buildrules
6. RESULTS
My Byte Hitrate for *.youtube.com has climbed from
about 3% to 30% over 8 days. I expect it to rise as
the number of youtube video objects in the cache
increases from a few hundreds to several thousands
over the next several months.
7.TODO
i. Update the script to expunge regexp rules for
expired objects.
ii. House Cleaning (which may never get done).
iii. A howto to share.
iv. Should the script be real time using tail -F? I
like watching videos a few more times after
Re: [squid-users] Caching Expired Objects - One Small Step Forward
On Sun, 7 Oct 2007, Solomon Asare wrote: Hi All, the long skelatel howto: 8. COMMENTS This is only one of many ways that this goal can be achieved and certailnly not the best being a non-guru, although a determined linux user. I guess the features I couldn't find in squid2.6-stable that made me add apache may be included in later stable releases to make this irrelevant. Unfortunately, I did not keep any logs whilst doing this so I amy have skipped a few steps. If I have, it will show sooner or later. I have tried to put together as much of the info that I think someone might need. The squid mailing list proved very helpful, and I am very grateful. There are many on the list prepared to help, although you may come accross a few who will repeatedly tell you how easy it is to do what you want to do without sharing how. Don't dispare if you bump into them. Great job solomon. Many of us have been trying to achieve similar with youtube and google vids. this will help a great deal. how big of a cachedir do you keep for youtube vids. should be quite a big to be able to cache the vids in large enough quantity to get a decent hit. I'm gonna try to achieve what you've describe here for my next project. Thanks for the job well done. Manoj Regards, solomon. --
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi, --- Manoj_Rajkarnikar [EMAIL PROTECTED] wrote: Great job solomon. Many of us have been trying to achieve similar with youtube and google vids. this will help a great deal. how big of a cachedir do you keep for youtube vids. should be quite a big to be able to cache the vids in large enough quantity to get a decent hit. I'm gonna try to achieve what you've describe here for my next project. Thanks for the job well done. Manoj Currently, my cache size is 70 GB. I have just filled almost 25 GB. I will upgrade to either 200 or 500 GB when I deploy. Solomon.
Re: [squid-users] Caching Expired Objects - One Small Step Forward
Hi All, a small success: I have set a an apache2.2 server as a proxy without caching as a parent proxy to my squid. I pass my youtube video clips through this parent. The apache is on the same machine as the squid. Using the mod_header module in apache, I cleaned out all the worrisome headers in the the youtube clips and now squid caches them and produces hits. The only outstanding issue is youtube serves videos from random servers. I will therefore have to implement a redirection script so that if a video clip has already been cached, I will pick that one. I hope to work on that over the weekend. I sometimes get about 10% traffic to youtube so if I can get a descent HIT rate it will be significant to the overall performance of my squid. Thanks for the hand-holding, Henrik. Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik
Re: [squid-users] Caching Expired Objects
On mån, 2007-09-24 at 07:42 -0700, zulkarnain wrote: I'm having the same problem, any solution to handle this? thanks! Solution: 0. Make sure the cache directive has been disabled. The distributed configuration file blocks caching of URLs with ? or cgi-bin in them.. 1. Understand why the object is not getting cached. This requires inspecting the HTTP headers looking for Cache-Control, Authorization and Pragma headers.. You can get access to the header details by enabling log_mime_hdrs in squid.conf (default off to protect users privacy). Also taking a close look at the URL is worthwhile as quite often there is unique identifiers in the URL changing for each request, making the URL unique.. to get access to query parameters disable strip_query_terms in squid.conf (default on to protect users privacy). 2. If the URL is static and it's just HTTP headers which prevents the object from being cached then override those reasons with suitable refresh_pattern settings. Or talk with the site owner and have them read http://www.mnot.net/cache_docs/ 3. If the URL is dynamic with some unique identifier it's more complex. May work stripping out the unique identifier using an url rewriter (see url_rewrite_program) but not guaranteed. Or talk with the site owner and have them read http://www.mnot.net/cache_docs/ Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
I'm having the same problem, any solution to handle this? thanks! Zul --- Solomon Asare [EMAIL PROTECTED] wrote: Hi All, sorry, the refresh_pattern looked like: refresh_pattern -i get\_video\?video\_id\=.*youtube\.com 10080 990% reload-into-ims ignore-no-cache override-expire ignore-private ignore-no-store Regards, solomon. --- Solomon Asare [EMAIL PROTECTED] wrote: Hi All, I have had some success in caching expired objects with squid3 as shown below: 1190548054.719 SWAPOUT 00 0018 55122D5A03E0C43DBA15AFDF9225215B 200-1 -1 1190548026 video/flv 1626399/1626399 GET http://74.125.0.25/get_video?video_id=tfYK0yBXS-oorigin=lax-v211.lax.youtube.com However I am unable to get a HIT with the same refresh_pattern that cached the object in the first place. Squid3client reports the object has expired: HTTP/1.0 403 Forbidden Server: squid/3.0.PRE5 Mime-Version: 1.0 Date: Sun, 23 Sep 2007 12:30:39 GMT Content-Type: text/html Content-Length: 1530 Expires: Sun, 23 Sep 2007 12:30:39 GMT X-Squid-Error: ERR_FORWARDING_DENIED 0 X-Cache: MISS from testSquid Via: 1.0 testSquid (squid/3.0.PRE5) Proxy-Connection: close Any suggestions? Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase. http://farechase.yahoo.com/
Re: [squid-users] Caching Expired Objects
Hi All, I have had some success in caching expired objects with squid3 as shown below: 1190548054.719 SWAPOUT 00 0018 55122D5A03E0C43DBA15AFDF9225215B 200-1 -1 1190548026 video/flv 1626399/1626399 GET http://74.125.0.25/get_video?video_id=tfYK0yBXS-oorigin=lax-v211.lax.youtube.com However I am unable to get a HIT with the same refresh_pattern that cached the object in the first place. Squid3client reports the object has expired: HTTP/1.0 403 Forbidden Server: squid/3.0.PRE5 Mime-Version: 1.0 Date: Sun, 23 Sep 2007 12:30:39 GMT Content-Type: text/html Content-Length: 1530 Expires: Sun, 23 Sep 2007 12:30:39 GMT X-Squid-Error: ERR_FORWARDING_DENIED 0 X-Cache: MISS from testSquid Via: 1.0 testSquid (squid/3.0.PRE5) Proxy-Connection: close Any suggestions? Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik
Re: [squid-users] Caching Expired Objects
Hi All, sorry, the refresh_pattern looked like: refresh_pattern -i get\_video\?video\_id\=.*youtube\.com 10080 990% reload-into-ims ignore-no-cache override-expire ignore-private ignore-no-store Regards, solomon. --- Solomon Asare [EMAIL PROTECTED] wrote: Hi All, I have had some success in caching expired objects with squid3 as shown below: 1190548054.719 SWAPOUT 00 0018 55122D5A03E0C43DBA15AFDF9225215B 200-1 -1 1190548026 video/flv 1626399/1626399 GET http://74.125.0.25/get_video?video_id=tfYK0yBXS-oorigin=lax-v211.lax.youtube.com However I am unable to get a HIT with the same refresh_pattern that cached the object in the first place. Squid3client reports the object has expired: HTTP/1.0 403 Forbidden Server: squid/3.0.PRE5 Mime-Version: 1.0 Date: Sun, 23 Sep 2007 12:30:39 GMT Content-Type: text/html Content-Length: 1530 Expires: Sun, 23 Sep 2007 12:30:39 GMT X-Squid-Error: ERR_FORWARDING_DENIED 0 X-Cache: MISS from testSquid Via: 1.0 testSquid (squid/3.0.PRE5) Proxy-Connection: close Any suggestions? Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik
Re: [squid-users] Caching Expired Objects
Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On tis, 2007-09-18 at 09:25 -0700, Solomon Asare wrote: Hi Henrik, since you say so, I have rather been toying with the idea of saving these supposedly expired objects in an apache document root and using the url_rewrite of the squid to fetch the objects from my apache server. I hope the bandwidth savings will justify the bandwidth cost in repopulating the apache with these objects. Its about bandwidth! That's pretty much the same as using refresh_pattern to give Squid a long freshness for those objects, or actually worse as you give the objects completely new HTTP meta information. If I were you I would use refresh_pattern, overriding the expiry information of these objects. Much less intrusive to the HTTP flow. Regards Henrik
Re: [squid-users] Caching Expired Objects
On ons, 2007-09-19 at 01:45 -0700, Solomon Asare wrote: Hi Henrik, I have tried quite a lot, eg: refresh_pattern -i \.flv$ 10080 990% 99 reload-into-ims ignore-no-cache It caches only those objects which have not already expired that is with the right combinations of Last-Modified or ETag minimum_expiry_time; as you explained earlier. Any suggestion on a refresh_pattern to overcome (Last-Modified or ETag minimum_expiry_time) limitation? Have you tried override-expire? What do use depends on what the response headers look like. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
Hi Henrik, --- Henrik Nordstrom [EMAIL PROTECTED] wrote: Have you tried override-expire? yes, I have tried overide-expire without success, as in: refresh_pattern -i \.flv$ 10080 990% 99 ignore-no-cache override-expire ignore-private Thanks, solomon.
Re: [squid-users] Caching Expired Objects
On ons, 2007-09-19 at 05:36 -0700, Solomon Asare wrote: yes, I have tried overide-expire without success, as in: refresh_pattern -i \.flv$ 10080 990% 99 ignore-no-cache override-expire ignore-private And what do the response headers look like for the object in question? Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
Hi Henrik, --- Henrik Nordstrom [EMAIL PROTECTED] wrote: And what do the response headers look like for the object in question? Some Header Info from my logs on the objects that will not cache: Keep-Alive: 300\r\nProxy-Connection: keep-alive\r\n] [HTTP/1.1 200 OK\r\nPragma: no-cache\r\nExpires: Sat, 17 Jan 1981 8:00:00 GMT\r\nCache-Control: no-store, no-cache, must-revalidate\r\nContent-Type: video/flv\r\nContent-Length: 7672205\r\n Connection: Keep-Alive\r\n] [HTTP/1.1 200 OK\r\nPragma: no-cache\r\nExpires: Sat, 17 Jan 1981 8:00:00 GMT\r\nCache-Control: no-store, no-cache, must-revalidate\r\nContent-Type: video/flv\r\nContent-Length: 7672205\r\n Are they air-tight or they can be managed? Regards, solomon. Regards Henrik
Re: [squid-users] Caching Expired Objects
On ons, 2007-09-19 at 12:40 -0700, Solomon Asare wrote: Connection: Keep-Alive\r\n] [HTTP/1.1 200 OK\r\nPragma: no-cache\r\nExpires: Sat, 17 Jan 1981 8:00:00 GMT\r\nCache-Control: no-store, no-cache, must-revalidate\r\nContent-Type: video/flv\r\nContent-Length: 7672205\r\n Are they air-tight or they can be managed? Lets se.. Pragma: no-cache- ignore-no-cache Expires: inthepast - override-expire Cache-Control: no-store - no override available Cache-Control: no-cache - ignore-no-cache Cache-Control: must-revalidate - only relevant when stale so yes, the server is very insisting on that the object should not be cached and there is not sufficient overrides available in Squid-2 to cache these (CC: no-store). To cache this you need to upgrade to Squid-3 where there is an ignore-no-store override option available. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
Hi Henrik, it looks hopeful, I will try your suggestion, Squid 3. Thanks again. I will report back. Already, just after deploying squid for about a week, I am saving 20% on banndwidth. Thanks, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On ons, 2007-09-19 at 12:40 -0700, Solomon Asare wrote: Connection: Keep-Alive\r\n] [HTTP/1.1 200 OK\r\nPragma: no-cache\r\nExpires: Sat, 17 Jan 1981 8:00:00 GMT\r\nCache-Control: no-store, no-cache, must-revalidate\r\nContent-Type: video/flv\r\nContent-Length: 7672205\r\n Are they air-tight or they can be managed? Lets se.. Pragma: no-cache - ignore-no-cache Expires: inthepast- override-expire Cache-Control: no-store - no override available Cache-Control: no-cache - ignore-no-cache Cache-Control: must-revalidate- only relevant when stale so yes, the server is very insisting on that the object should not be cached and there is not sufficient overrides available in Squid-2 to cache these (CC: no-store). To cache this you need to upgrade to Squid-3 where there is an ignore-no-store override option available. Regards Henrik
Re: [squid-users] Caching Expired Objects
Hi Henrik, thanks for your insightful response. However, the object is a .flv file that hasn't changed in months. The origin server certainly doesn't want the object cached, but I want to. Any leads that can help me achieve this? Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On mån, 2007-09-17 at 11:55 -0700, Solomon Asare wrote: Hi Amos, I am not sure if refresh_pattern is the sole determinant in caching an object, that is if it has any influence at all. It has influence, both directly by assigning freshness information when there is none, and indirectly by overriding various HTTP controls.. Requirementsto cache stale objects: a) The object must have a cache validator (Last-Modified or ETag). If there is no cache validator then the response must be fresh for at least minimum_expiry_time to get cached, this to avoid wasting disk I/O for caching content which can not be reused. b) There must not be other headers preventing it from getting cached. refresh_pattern can override most of these if needed. I am not discussing getting a HIT for a cached object, but rather caching an expired object from an origin server. If this object is expired, by say 60 seconds before being served from the origin server, how do I cache it? Date and Last-Modified dates are also not set. If there is no Last-Modified and no ETag then it's useless to cache an expired object, as it can not be reused on any future request and all you get is extra disk I/O for writing the object out. A cache validator (Last-Modified or ETag) is required to be able to verify with the origin server if an expired object is still valid or not. Without a cache validator there is nothing to relate to and there is no other choice than to fetch the complete object again when expired.. Regards Henrik
Re: [squid-users] Caching Expired Objects
On Tue, Sep 18, 2007, Solomon Asare wrote: Hi Henrik, thanks for your insightful response. However, the object is a .flv file that hasn't changed in months. The origin server certainly doesn't want the object cached, but I want to. Any leads that can help me achieve this? * set your refresh_pattern's right, you can override almost all the relevant headers in there; * if the URL has a ? in it then you need to look at the cache/no_cache directives * if in doubt, compile with the option to log request/reply headers (I forget what it is, ./configure --help will tell you) and take a look at exactly what headers they're sending back. Adrian Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On m?n, 2007-09-17 at 11:55 -0700, Solomon Asare wrote: Hi Amos, I am not sure if refresh_pattern is the sole determinant in caching an object, that is if it has any influence at all. It has influence, both directly by assigning freshness information when there is none, and indirectly by overriding various HTTP controls.. Requirementsto cache stale objects: a) The object must have a cache validator (Last-Modified or ETag). If there is no cache validator then the response must be fresh for at least minimum_expiry_time to get cached, this to avoid wasting disk I/O for caching content which can not be reused. b) There must not be other headers preventing it from getting cached. refresh_pattern can override most of these if needed. I am not discussing getting a HIT for a cached object, but rather caching an expired object from an origin server. If this object is expired, by say 60 seconds before being served from the origin server, how do I cache it? Date and Last-Modified dates are also not set. If there is no Last-Modified and no ETag then it's useless to cache an expired object, as it can not be reused on any future request and all you get is extra disk I/O for writing the object out. A cache validator (Last-Modified or ETag) is required to be able to verify with the origin server if an expired object is still valid or not. Without a cache validator there is nothing to relate to and there is no other choice than to fetch the complete object again when expired.. Regards Henrik -- - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support - - $25/pm entry-level bandwidth-capped VPSes available in WA -
Re: [squid-users] Caching Expired Objects
On tis, 2007-09-18 at 02:55 -0700, Solomon Asare wrote: This is the exact problem I have that I am trying to resolve, not querry string issues. If only I can overide the lack of Last-Modified, Etag and not meeting minimum_expiry_time conditions. There would be no use doing so. All you would get is more disk I/O as Squid would be unable to reuse the cached copy on the next request. Without a cache validator you MUST assign freshness to the object for it to be of any use. Think of it, what do you want Squid to do with the expired object if it can not check if the object has changed (validator required), and you do not allow it to consider the object as fresh? Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
Hi Henrik, since you say so, I have rather been toying with the idea of saving these supposedly expired objects in an apache document root and using the url_rewrite of the squid to fetch the objects from my apache server. I hope the bandwidth savings will justify the bandwidth cost in repopulating the apache with these objects. Its about bandwidth! Regards, solomon. --- Henrik Nordstrom [EMAIL PROTECTED] wrote: On tis, 2007-09-18 at 02:55 -0700, Solomon Asare wrote: This is the exact problem I have that I am trying to resolve, not querry string issues. If only I can overide the lack of Last-Modified, Etag and not meeting minimum_expiry_time conditions. There would be no use doing so. All you would get is more disk I/O as Squid would be unable to reuse the cached copy on the next request. Without a cache validator you MUST assign freshness to the object for it to be of any use. Think of it, what do you want Squid to do with the expired object if it can not check if the object has changed (validator required), and you do not allow it to consider the object as fresh? Regards Henrik
Re: [squid-users] Caching Expired Objects
On tis, 2007-09-18 at 09:25 -0700, Solomon Asare wrote: Hi Henrik, since you say so, I have rather been toying with the idea of saving these supposedly expired objects in an apache document root and using the url_rewrite of the squid to fetch the objects from my apache server. I hope the bandwidth savings will justify the bandwidth cost in repopulating the apache with these objects. Its about bandwidth! That's pretty much the same as using refresh_pattern to give Squid a long freshness for those objects, or actually worse as you give the objects completely new HTTP meta information. If I were you I would use refresh_pattern, overriding the expiry information of these objects. Much less intrusive to the HTTP flow. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Caching Expired Objects
On Mon, Sep 17, 2007, Solomon Asare wrote: Hi All, please, to recast, how do I cache expired objects. I get responses with expiry dates being in the past from some servers that I will want to cache. refresh_pattern adrian
Re: [squid-users] Caching Expired Objects
Hi, just to clarify: I don't mean getting a HIT, just caching the object into my store. Any link to a refresh_pattern that will cache an object that has expired for about 60 secs by the time you fetch it? Regards, solomon. --- Adrian Chadd [EMAIL PROTECTED] wrote: On Mon, Sep 17, 2007, Solomon Asare wrote: Hi All, please, to recast, how do I cache expired objects. I get responses with expiry dates being in the past from some servers that I will want to cache. refresh_pattern adrian
Re: [squid-users] Caching Expired Objects
Solomon Asare wrote: Hi, just to clarify: I don't mean getting a HIT, just caching the object into my store. Any link to a refresh_pattern that will cache an object that has expired for about 60 secs by the time you fetch it? http://www.squid-cache.org/Versions/v2/2.6/cfgman/refresh_pattern.html http://www.squid-cache.org/Versions/v3/3.0/cfgman/refresh_pattern.html Amos
