Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-10-16 Thread BUI18
Sorry it took a while to get back.  Not sure how to interpre X-Cache and 
X-Cache-Lookup.

Here's the header info from Fiddler:

Request Header

GET /server1/websites/data/folder/myvideofile.vid HTTP/1.1

Client
Accept: */*
Transport
Host: ftp.mydomain.com
Proxy-Connection: Keep-Alive

Response Header

HTTP/1.0 200 OK
Content-Length: 1775372
Content-Type: video/jpeg
Last-Modified: Tue, 02 Sep 2008 23:57:25 GMT
Accept-Ranges: none
ETag: 8020b2a557dc91:3ecc
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Thu, 16 Oct 2008 22:52:49 GMT
X-Cache: MISS from squid.mydomain.com
X-Cache-Lookup: HIT from squid.mydomain.com:3128
Via: 1.0 squid.mydomain.com:3128 (squid/2.6.STABLE14)
Proxy-Connection: keep-alive





- Original Message 
From: Amos Jeffries [EMAIL PROTECTED]
To: BUI18 [EMAIL PROTECTED]
Cc: squid-users@squid-cache.org; Itzcak Pechtalt [EMAIL PROTECTED]
Sent: Wednesday, September 24, 2008 6:17:22 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

BUI18 wrote:
 Hi - Thanks for responding.  URL for video file never changes.
 

What release of Squid?

Did you check the Expires header properly from the transfer rather than 
from the (apparently untrustworthy) info in the store log?


 I did some more checking in the Squid logs and this is what I noticed:
 
 File Properties of video file (Pacific Daylight Time (PDT))
 
 Created On: Monday, September 22, 2008, 8:59:35 AM
 
 Modified On: Monday, September 22, 2008, 8:59:35 AM
 
 Accessed On: Today, September 24, 2008, 3:53:12 AM
 
 ***
 Wget Grabs File (Time in India Standard Time (IST))
 
 --04:38:35--  http://ftp.mydomain.com/websites/data/myvideofile.vid
  = `/WGET/Temp/myvideofile.vid'
 04:38:54 (93.91 KB/s) - `/WGET/Temp/myvideofile.vid' saved [1791244/1791244]
 
 The access.log confirms initial pre-fetch by wget.
 
 1222124934.241  18968 192.168.200.4 TCP_MISS/200 1791684 GET 
 http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
 video/jpeg
 
 UTC = Mon, 22 Sep 2008 23:08:54 GMT
 
 The store.log shows a write from memory to disk:
 
 1222124934.241 SWAPOUT 00 00057B65 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
 1222124765 1222099175-1 video/jpeg 1791244/1791244 GET 
 http://ftp.mydomain.com/websites/data/myvideofile.vid
 
 UTC = Mon, 22 Sep 2008 23:08:54 GMT
 
 ***
 
 Then Store.log shows release or removal from cache:
 
 153725.068 RELEASE 00 00057B65 605FAC36E93B0CDE81902BBC6C5EC71A  200 
 1222124765 1222099175-1 video/jpeg 1791244/-279 GET 
 http://ftp.mydomain.com/websites/data/myvideofile.vid
 
 UTC = Wed, 24 Sep 2008 10:55:25 GMT
 
 Notice the -1 for expiration header (I do not set one on the object).  My min 
 age is 5 days so I'm not sure why the object would be released from cache in 
 less than 2 days.
 
 If the object was released from cache, when the user tried to access file, 
 Squid reports TCP_REFRESH_MISS, which to me means that it was found in cache 
 but when it sends a If-Modified-Since request, it thinks that the file has 
 been modified (which it was not as seen by the lastmod date indicated in the 
 store.log below.
 
 ***
 
 User accessed file (access.log):
 
 153742.005  17275 192.168.200.52 TCP_REFRESH_MISS/200 1791688 GET 
 http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
 video/jpeg
 
 UTC = Wed, 24 Sep 2008 10:55:42 GMT
 
 Then store.log shows a write to disk
 
 153742.005 SWAPOUT 00 00088336 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
 153575 1222099175-1 video/jpeg 
 1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid
 
 UTC = Wed, 24 Sep 2008 10:55:42 GMT
 datehdr: Wed, 24 Sep 2008 10:55:55 GMT
 lastmod: Mon, 22 Sep 2008 15:59:35 GMT
 
 Anyone with ideas on why this behavior occurs?
 
 thanks
 
 
 
 
 
 - Original Message 
 From: Itzcak Pechtalt [EMAIL PROTECTED]
 To: Squid Users squid-users@squid-cache.org
 Sent: Wednesday, September 24, 2008 4:35:59 AM
 Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max
 
 On Wed, Sep 24, 2008 at 1:39 PM, BUI18 [EMAIL PROTECTED] wrote:
 Hi -

 I have squid box with tons of disk for the cache_dir
 (hundreds of GB).  I use wget to perform some pre-fetching of large
 video files.  I've set the min and max age to 5 days and 7 days (in
 minutes).  And although I have plenty of disk space available, I still
 receive TCP_REFRESH_MISS for files that had been pre-fetched and later
 accessed the same day.  Does anyone know why Squid would consider it as
 STALE?  I thought that by setting the min value for refresh_pattern for
 the video file would guarantee freshness.  Not only does the cache
 consider it STALE, it then goes and pre-fetches a new copy even though
 I know

Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-26 Thread Adrian Chadd
Well, what are the complete request/reply headers for each of the
requests you're testing with?


Adrian

2008/9/25 BUI18 [EMAIL PROTECTED]:
 My Squid Version is 2.6/STABLE14

 Here's my refresh_pattern from squid.conf

 #Suggested default:
 refresh_pattern ^ftp:   144020% 10080
 refresh_pattern ^gopher:14400%  1440

 #The following line will ignore a client no-cache header
 #refresh_pattern -i \.vid$   0   90% 2880 ignore-reload
 refresh_pattern -i \.vid$   7200100%10080 ignore-reload

 refresh_pattern .   0   20% 4320

 A link to the file looks something like this -- 
 http://ftp.mydomain.com/websites/data/myvideofile.vid

 I have to set up a station to grab the header but I can tell you that it does 
 not seem out of the ordinary.

 There is one cache-control:  Pragma: no-cache

 I believe I handle this with the ignore-reload options.

 Our server is an IIS server running on Windows 2003.

 I also ran a test with min and max age of 0 and 1 respectively, and it seems 
 to work.  I receive a TCP_REFRESH_HIT, which is what I would have expected as 
 these files do not change.

 Please let me know if you have any other ideas on how to track down why it 
 would release from cache before min age with no Expiration set on the object.

 Open to any suggestions.
 Thanks




 - Original Message 
 From: Michael Alger [EMAIL PROTECTED]
 To: squid-users@squid-cache.org
 Sent: Wednesday, September 24, 2008 8:09:50 AM
 Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

 On Wed, Sep 24, 2008 at 05:29:52AM -0700, BUI18 wrote:
 I went through your same thinking as you described below.

 I checked the Expires header from the server and we do not set
 one.  I checked via Fiddler web debug tool.  I also verified with
 the dev guys here regarding no Expires header.  I have set the min
 and max via refresh_pattern because of the absence of the Expires
 header thinking that Squid would keep it FRESH.

 Notice the -1 for expiration header (I do not set one on the
 object).  My min age is 5 days so I'm not sure why the object
 would be released from cache in less than 2 days.

 If the object was released from cache, when the user tried to
 access file, Squid reports TCP_REFRESH_MISS, which to me means
 that it was found in cache but when it sends a If-Modified-Since
 request, it thinks that the file has been modified (which it was
 not as seen by the lastmod date indicated in the store.log below.

 Interesting that it's caching the file for 2 days. What are the full
 headers returned with the object? Any other cache control headers?

 Is there any chance you have a conflicting refresh_pattern, so the
 freshness rules being applied aren't the ones you're expecting? May
 be worth doing some tests with very small max ages to confirm it's
 matching the right rule.








[squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread BUI18
Hi -

I have squid box with tons of disk for the cache_dir
(hundreds of GB).  I use wget to perform some pre-fetching of large
video files.  I've set the min and max age to 5 days and 7 days (in
minutes).  And although I have plenty of disk space available, I still
receive TCP_REFRESH_MISS for files that had been pre-fetched and later
accessed the same day.  Does anyone know why Squid would consider it as
STALE?  I thought that by setting the min value for refresh_pattern for
the video file would guarantee freshness.  Not only does the cache
consider it STALE, it then goes and pre-fetches a new copy even though
I know that the video file has not changed.  Any help would be greatly
appreciated.  Thanks.


  


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread Itzcak Pechtalt
On Wed, Sep 24, 2008 at 1:39 PM, BUI18 [EMAIL PROTECTED] wrote:
 Hi -

 I have squid box with tons of disk for the cache_dir
 (hundreds of GB).  I use wget to perform some pre-fetching of large
 video files.  I've set the min and max age to 5 days and 7 days (in
 minutes).  And although I have plenty of disk space available, I still
 receive TCP_REFRESH_MISS for files that had been pre-fetched and later
 accessed the same day.  Does anyone know why Squid would consider it as
 STALE?  I thought that by setting the min value for refresh_pattern for
 the video file would guarantee freshness.  Not only does the cache
 consider it STALE, it then goes and pre-fetches a new copy even though
 I know that the video file has not changed.  Any help would be greatly
 appreciated.  Thanks.





Hi,
Check if the video URL changes from request to request. In YouTube
video even if the main URL is the same, there is request ID in URL who
changes per request.

Itzcak


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread Michael Alger
On Wed, Sep 24, 2008 at 03:39:16AM -0700, BUI18 wrote:
 I have squid box with tons of disk for the cache_dir
 (hundreds of GB).  I use wget to perform some pre-fetching of large
 video files.  I've set the min and max age to 5 days and 7 days (in
 minutes).  And although I have plenty of disk space available, I still
 receive TCP_REFRESH_MISS for files that had been pre-fetched and later
 accessed the same day.  Does anyone know why Squid would consider it as
 STALE?  I thought that by setting the min value for refresh_pattern for
 the video file would guarantee freshness.  Not only does the cache
 consider it STALE, it then goes and pre-fetches a new copy even though
 I know that the video file has not changed.  Any help would be greatly
 appreciated.  Thanks.

The fact that it's doing TCP_REFRESH_xxx means squid does have a
cached copy which it considers potentially stale. So it's sending an
If-Modified-Since request to the origin server. The origin is then
either saying yes, it's been modified since you retrieved it --
here's a new one; or it has no idea how to handle IMS and is
sending the whole object regardless.

What Expires: header is the server sending? You can use the -S
switch with wget to show the server response headers when you're
doing your pre-fetch. If they look fine, maybe keep a copy of them
and compare later in the day when squid decides it needs a new one.

I assume you're setting the min and max age via refresh_pattern
lines? Remember that these are only used in the absence of an
Expires header.


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread BUI18
Hi - Thanks for responding.  URL for video file never changes.

I did some more checking in the Squid logs and this is what I noticed:

File Properties of video file (Pacific Daylight Time (PDT))

Created On: Monday, September 22, 2008, 8:59:35 AM

Modified On: Monday, September 22, 2008, 8:59:35 AM

Accessed On: Today, September 24, 2008, 3:53:12 AM

***
Wget Grabs File (Time in India Standard Time (IST))

--04:38:35--  http://ftp.mydomain.com/websites/data/myvideofile.vid
 = `/WGET/Temp/myvideofile.vid'
04:38:54 (93.91 KB/s) - `/WGET/Temp/myvideofile.vid' saved [1791244/1791244]

The access.log confirms initial pre-fetch by wget.

1222124934.241  18968 192.168.200.4 TCP_MISS/200 1791684 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Mon, 22 Sep 2008 23:08:54 GMT

The store.log shows a write from memory to disk:

1222124934.241 SWAPOUT 00 00057B65 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
1222124765 1222099175-1 video/jpeg 1791244/1791244 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Mon, 22 Sep 2008 23:08:54 GMT

***

Then Store.log shows release or removal from cache:

153725.068 RELEASE 00 00057B65 605FAC36E93B0CDE81902BBC6C5EC71A  200 
1222124765 1222099175-1 video/jpeg 1791244/-279 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:25 GMT

Notice the -1 for expiration header (I do not set one on the object).  My min 
age is 5 days so I'm not sure why the object would be released from cache in 
less than 2 days.

If the object was released from cache, when the user tried to access file, 
Squid reports TCP_REFRESH_MISS, which to me means that it was found in cache 
but when it sends a If-Modified-Since request, it thinks that the file has been 
modified (which it was not as seen by the lastmod date indicated in the 
store.log below.

***

User accessed file (access.log):

153742.005  17275 192.168.200.52 TCP_REFRESH_MISS/200 1791688 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Wed, 24 Sep 2008 10:55:42 GMT

Then store.log shows a write to disk

153742.005 SWAPOUT 00 00088336 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
153575 1222099175-1 video/jpeg 
1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:42 GMT
datehdr: Wed, 24 Sep 2008 10:55:55 GMT
lastmod: Mon, 22 Sep 2008 15:59:35 GMT

Anyone with ideas on why this behavior occurs?

thanks





- Original Message 
From: Itzcak Pechtalt [EMAIL PROTECTED]
To: Squid Users squid-users@squid-cache.org
Sent: Wednesday, September 24, 2008 4:35:59 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

On Wed, Sep 24, 2008 at 1:39 PM, BUI18 [EMAIL PROTECTED] wrote:
 Hi -

 I have squid box with tons of disk for the cache_dir
 (hundreds of GB).  I use wget to perform some pre-fetching of large
 video files.  I've set the min and max age to 5 days and 7 days (in
 minutes).  And although I have plenty of disk space available, I still
 receive TCP_REFRESH_MISS for files that had been pre-fetched and later
 accessed the same day.  Does anyone know why Squid would consider it as
 STALE?  I thought that by setting the min value for refresh_pattern for
 the video file would guarantee freshness.  Not only does the cache
 consider it STALE, it then goes and pre-fetches a new copy even though
 I know that the video file has not changed.  Any help would be greatly
 appreciated.  Thanks.





Hi,
Check if the video URL changes from request to request. In YouTube
video even if the main URL is the same, there is request ID in URL who
changes per request.

Itzcak



  


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread BUI18
Hi -

I went through your same thinking as you described below.

I checked the Expires header from the server and we do not set one.  I checked 
via Fiddler web debug tool.  I also verified with the dev guys here regarding 
no Expires header.  I have set the min and max via refresh_pattern because of 
the absence of the Expires header thinking that Squid would keep it FRESH.

I recently posted the details of the Squid logs, but will re-post it here so 
that the thread will follow properly (it may help others when search).

Hoping someone could explain the behavior I see below.


File Properties of video file (Pacific Daylight Time (PDT))

Created On: Monday, September 22, 2008, 8:59:35 AM

Modified On: Monday, September 22, 2008, 8:59:35 AM

Accessed On: Today, September 24, 2008, 3:53:12 AM

***
Wget Grabs File (Time in India Standard Time (IST))

--04:38:35--  http://ftp.mydomain.com/websites/data/myvideofile.vid
= `/WGET/Temp/myvideofile.vid'
04:38:54 (93.91 KB/s) - `/WGET/Temp/myvideofile.vid' saved [1791244/1791244]

The access.log confirms initial pre-fetch by wget.

1222124934.241  18968 192.168.200.4 TCP_MISS/200 1791684 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Mon, 22 Sep 2008 23:08:54 GMT

The store.log shows a write from memory to disk:

1222124934.241 SWAPOUT 00 00057B65 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
1222124765 1222099175-1 video/jpeg 1791244/1791244 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Mon, 22 Sep 2008 23:08:54 GMT

***

Then Store.log shows release or removal from cache:

153725.068 RELEASE 00 00057B65 605FAC36E93B0CDE81902BBC6C5EC71A  200 
1222124765 1222099175-1 video/jpeg 1791244/-279 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:25 GMT

Notice
the -1 for expiration header (I do not set one on the object).  My min
age is 5 days so I'm not sure why the object would be released from
cache in less than 2 days.

If the object was released from
cache, when the user tried to access file, Squid reports
TCP_REFRESH_MISS, which to me means that it was found in cache but when
it sends a If-Modified-Since request, it thinks that the file has been
modified (which it was not as seen by the lastmod date indicated in the
store.log below.

***

User accessed file (access.log):

153742.005  17275 192.168.200.52 TCP_REFRESH_MISS/200 1791688 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Wed, 24 Sep 2008 10:55:42 GMT

Then store.log shows a write to disk

153742.005 SWAPOUT 00 00088336 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
153575 1222099175-1 video/jpeg 
1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:42 GMT
datehdr: Wed, 24 Sep 2008 10:55:55 GMT
lastmod: Mon, 22 Sep 2008 15:59:35 GMT






- Original Message 
From: Michael Alger [EMAIL PROTECTED]
To: squid-users@squid-cache.org
Sent: Wednesday, September 24, 2008 4:49:38 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

On Wed, Sep 24, 2008 at 03:39:16AM -0700, BUI18 wrote:
 I have squid box with tons of disk for the cache_dir
 (hundreds of GB).  I use wget to perform some pre-fetching of large
 video files.  I've set the min and max age to 5 days and 7 days (in
 minutes).  And although I have plenty of disk space available, I still
 receive TCP_REFRESH_MISS for files that had been pre-fetched and later
 accessed the same day.  Does anyone know why Squid would consider it as
 STALE?  I thought that by setting the min value for refresh_pattern for
 the video file would guarantee freshness.  Not only does the cache
 consider it STALE, it then goes and pre-fetches a new copy even though
 I know that the video file has not changed.  Any help would be greatly
 appreciated.  Thanks.

The fact that it's doing TCP_REFRESH_xxx means squid does have a
cached copy which it considers potentially stale. So it's sending an
If-Modified-Since request to the origin server. The origin is then
either saying yes, it's been modified since you retrieved it --
here's a new one; or it has no idea how to handle IMS and is
sending the whole object regardless.

What Expires: header is the server sending? You can use the -S
switch with wget to show the server response headers when you're
doing your pre-fetch. If they look fine, maybe keep a copy of them
and compare later in the day when squid decides it needs a new one.

I assume you're setting the min and max age via refresh_pattern
lines? Remember that these are only used in the absence of an
Expires header.



  


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread Amos Jeffries

BUI18 wrote:

Hi - Thanks for responding.  URL for video file never changes.



What release of Squid?

Did you check the Expires header properly from the transfer rather than 
from the (apparently untrustworthy) info in the store log?




I did some more checking in the Squid logs and this is what I noticed:

File Properties of video file (Pacific Daylight Time (PDT))

Created On: Monday, September 22, 2008, 8:59:35 AM

Modified On: Monday, September 22, 2008, 8:59:35 AM

Accessed On: Today, September 24, 2008, 3:53:12 AM

***
Wget Grabs File (Time in India Standard Time (IST))

--04:38:35--  http://ftp.mydomain.com/websites/data/myvideofile.vid
 = `/WGET/Temp/myvideofile.vid'
04:38:54 (93.91 KB/s) - `/WGET/Temp/myvideofile.vid' saved [1791244/1791244]

The access.log confirms initial pre-fetch by wget.

1222124934.241  18968 192.168.200.4 TCP_MISS/200 1791684 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Mon, 22 Sep 2008 23:08:54 GMT

The store.log shows a write from memory to disk:

1222124934.241 SWAPOUT 00 00057B65 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
1222124765 1222099175-1 video/jpeg 1791244/1791244 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Mon, 22 Sep 2008 23:08:54 GMT

***

Then Store.log shows release or removal from cache:

153725.068 RELEASE 00 00057B65 605FAC36E93B0CDE81902BBC6C5EC71A  200 
1222124765 1222099175-1 video/jpeg 1791244/-279 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:25 GMT

Notice the -1 for expiration header (I do not set one on the object).  My min 
age is 5 days so I'm not sure why the object would be released from cache in 
less than 2 days.

If the object was released from cache, when the user tried to access file, 
Squid reports TCP_REFRESH_MISS, which to me means that it was found in cache 
but when it sends a If-Modified-Since request, it thinks that the file has been 
modified (which it was not as seen by the lastmod date indicated in the 
store.log below.

***

User accessed file (access.log):

153742.005  17275 192.168.200.52 TCP_REFRESH_MISS/200 1791688 GET 
http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 
video/jpeg

UTC = Wed, 24 Sep 2008 10:55:42 GMT

Then store.log shows a write to disk

153742.005 SWAPOUT 00 00088336 1E18E35BDC9307C6BC3FBEFD5B4120A3  200 
153575 1222099175-1 video/jpeg 
1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:42 GMT
datehdr: Wed, 24 Sep 2008 10:55:55 GMT
lastmod: Mon, 22 Sep 2008 15:59:35 GMT


Anyone with ideas on why this behavior occurs?

thanks





- Original Message 
From: Itzcak Pechtalt [EMAIL PROTECTED]
To: Squid Users squid-users@squid-cache.org
Sent: Wednesday, September 24, 2008 4:35:59 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

On Wed, Sep 24, 2008 at 1:39 PM, BUI18 [EMAIL PROTECTED] wrote:

Hi -

I have squid box with tons of disk for the cache_dir
(hundreds of GB).  I use wget to perform some pre-fetching of large
video files.  I've set the min and max age to 5 days and 7 days (in
minutes).  And although I have plenty of disk space available, I still
receive TCP_REFRESH_MISS for files that had been pre-fetched and later
accessed the same day.  Does anyone know why Squid would consider it as
STALE?  I thought that by setting the min value for refresh_pattern for
the video file would guarantee freshness.  Not only does the cache
consider it STALE, it then goes and pre-fetches a new copy even though
I know that the video file has not changed.  Any help would be greatly
appreciated.  Thanks.






Hi,
Check if the video URL changes from request to request. In YouTube
video even if the main URL is the same, there is request ID in URL who
changes per request.

Itzcak



  



--
Please use Squid 2.7.STABLE4 or 3.0.STABLE9


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread Michael Alger
On Wed, Sep 24, 2008 at 05:29:52AM -0700, BUI18 wrote:
 I went through your same thinking as you described below.
 
 I checked the Expires header from the server and we do not set
 one.  I checked via Fiddler web debug tool.  I also verified with
 the dev guys here regarding no Expires header.  I have set the min
 and max via refresh_pattern because of the absence of the Expires
 header thinking that Squid would keep it FRESH.
 
 Notice the -1 for expiration header (I do not set one on the
 object).  My min age is 5 days so I'm not sure why the object
 would be released from cache in less than 2 days.
 
 If the object was released from cache, when the user tried to
 access file, Squid reports TCP_REFRESH_MISS, which to me means
 that it was found in cache but when it sends a If-Modified-Since
 request, it thinks that the file has been modified (which it was
 not as seen by the lastmod date indicated in the store.log below.

Interesting that it's caching the file for 2 days. What are the full
headers returned with the object? Any other cache control headers?

Is there any chance you have a conflicting refresh_pattern, so the
freshness rules being applied aren't the ones you're expecting? May
be worth doing some tests with very small max ages to confirm it's
matching the right rule.


Re: [squid-users] Object becomes STALE: refresh_pattern min and max

2008-09-24 Thread BUI18
My Squid Version is 2.6/STABLE14

Here's my refresh_pattern from squid.conf

#Suggested default:
refresh_pattern ^ftp:   144020% 10080
refresh_pattern ^gopher:14400%  1440

#The following line will ignore a client no-cache header
#refresh_pattern -i \.vid$   0   90% 2880 ignore-reload
refresh_pattern -i \.vid$   7200100%10080 ignore-reload

refresh_pattern .   0   20% 4320

A link to the file looks something like this -- 
http://ftp.mydomain.com/websites/data/myvideofile.vid

I have to set up a station to grab the header but I can tell you that it does 
not seem out of the ordinary.

There is one cache-control:  Pragma: no-cache

I believe I handle this with the ignore-reload options.

Our server is an IIS server running on Windows 2003.

I also ran a test with min and max age of 0 and 1 respectively, and it seems to 
work.  I receive a TCP_REFRESH_HIT, which is what I would have expected as 
these files do not change.

Please let me know if you have any other ideas on how to track down why it 
would release from cache before min age with no Expiration set on the object.

Open to any suggestions.
Thanks




- Original Message 
From: Michael Alger [EMAIL PROTECTED]
To: squid-users@squid-cache.org
Sent: Wednesday, September 24, 2008 8:09:50 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

On Wed, Sep 24, 2008 at 05:29:52AM -0700, BUI18 wrote:
 I went through your same thinking as you described below.
 
 I checked the Expires header from the server and we do not set
 one.  I checked via Fiddler web debug tool.  I also verified with
 the dev guys here regarding no Expires header.  I have set the min
 and max via refresh_pattern because of the absence of the Expires
 header thinking that Squid would keep it FRESH.
 
 Notice the -1 for expiration header (I do not set one on the
 object).  My min age is 5 days so I'm not sure why the object
 would be released from cache in less than 2 days.
 
 If the object was released from cache, when the user tried to
 access file, Squid reports TCP_REFRESH_MISS, which to me means
 that it was found in cache but when it sends a If-Modified-Since
 request, it thinks that the file has been modified (which it was
 not as seen by the lastmod date indicated in the store.log below.

Interesting that it's caching the file for 2 days. What are the full
headers returned with the object? Any other cache control headers?

Is there any chance you have a conflicting refresh_pattern, so the
freshness rules being applied aren't the ones you're expecting? May
be worth doing some tests with very small max ages to confirm it's
matching the right rule.