Re: [squid-users] Objects Release from Cache Earlier Than Expected
After some further investigation, it seem that RELEASE does not mean that Squid deletes the object from cache. It appears that it releases from cache to the request. To restate the problem I am having: Squid seems to re-fetch the entire object even though the object never changed on the server after about 1 day - 2 days. Here's my test scenario: Object is initially cached. Max age in squid.conf is set to 1 min. Before 1 min passes, I request the object and Squid returns TCP_HIT. After 1 min, I try to request for object again. Squid returns TCP_REFRESH_HIT, which is what I expect. I leave the entire system untouched. A day or a day and a half later, I ask for the object again and Squid returns TCP_REFRESH_MISS/200. What could possibly cause Squid to refetch the entire object again? Could there possibly be a problem with the interaction between IE7 and Squid that is forcing Squid to re-fetch the entire object? Anyone with ideas on why this behavior occurs? Thanks - Original Message From: BUI18 [EMAIL PROTECTED] To: Henrik Nordstrom [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Tuesday, October 21, 2008 8:25:14 AM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected The web server is IIS 6. 1) Would there be any reason why it would return the full object when in fact the object has not been modified? 2) If the min age guarantees freshness of the object, why would Squid actually issue and IMS request to the web server in the first place? As I understand it, Squid should only issue and IMS request when objects become STALE. As such, I would have expected Squid to return TCP_HIT instead of TCP_REFRESH_MISS. 3) My big concern is that the store.log shows that the object was released (deleted) from cache well before the min age while there is still and abundant amount of disk space is available. Also, one other question: When Squid issues and IMS request, which date does it use? Is it the date/time that it retrieved the object or is it the Last Modified date/time of the object ascertained by Squid on first retrieval of the object? Regards -bui - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Tuesday, October 21, 2008 3:50:20 AM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On mån, 2008-10-20 at 17:45 -0700, BUI18 wrote: I not sure what you mean by a newer copy of the same URL? Can you elaborate on that a bit? The cache (i.e. Squid) performed a conditional request to the origin web server, and the web server returned a new 200 OK object with full content instead of a small 304 Not Modified. Regards Henrik
Re: [squid-users] Objects Release from Cache Earlier Than Expected
On ons, 2008-10-22 at 14:35 -0700, BUI18 wrote: Object is initially cached. Max age in squid.conf is set to 1 min. Before 1 min passes, I request the object and Squid returns TCP_HIT. After 1 min, I try to request for object again. Squid returns TCP_REFRESH_HIT, which is what I expect. I leave the entire system untouched. A day or a day and a half later, I ask for the object again and Squid returns TCP_REFRESH_MISS/200. TCP_HIT is a local hit on the Squid cache. Origin server was not asked. TCP_REFRESH_HIT is a cache hit after the origin server was asked if the object is still fresh. TCP_REFREHS_MISS is when the origin server says the object is no longer fresh and returns a new copy on the conditional query sent by the cache. (same query as in TCP_REFRESH_HIT, different response from the web server). What could possibly cause Squid to refetch the entire object again? A better question is why your server responds with the entire object on a If-Modified-Since type query if it hasn't been modified. It should have responded with a 304 response as it did in the TCP_REFRESH_HIT case. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Objects Release from Cache Earlier Than Expected
Henrik - Thanks for taking time out to respond to my questions. I'm completely stumped on this one. In our production environment, we set min and max to 5 and 7 days, respectively. As I understand it, if the request is made for the object in say3 days or 4 days (less than 5 days), I would always expect a TCP_HIT. But again, after 1 to 2 days, I see TCP_REFRESH_MISS and I get the whole object. I thought that by setting the min to 5 days would guarantee freshness up to 5 days. Do you know of a problem that maybe causes Squid to ignore the rules on determining whether an object is fresh? We used fiddler and actually removed the If-Modified-Since part of the request and still we get TCP_REFRESH_MISS. Do you have any other ideas on areas we might want to check to see what could possibly be causing this behavior? Thanks - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Wednesday, October 22, 2008 4:06:33 PM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On ons, 2008-10-22 at 14:35 -0700, BUI18 wrote: Object is initially cached. Max age in squid.conf is set to 1 min. Before 1 min passes, I request the object and Squid returns TCP_HIT. After 1 min, I try to request for object again. Squid returns TCP_REFRESH_HIT, which is what I expect. I leave the entire system untouched. A day or a day and a half later, I ask for the object again and Squid returns TCP_REFRESH_MISS/200. TCP_HIT is a local hit on the Squid cache. Origin server was not asked. TCP_REFRESH_HIT is a cache hit after the origin server was asked if the object is still fresh. TCP_REFREHS_MISS is when the origin server says the object is no longer fresh and returns a new copy on the conditional query sent by the cache. (same query as in TCP_REFRESH_HIT, different response from the web server). What could possibly cause Squid to refetch the entire object again? A better question is why your server responds with the entire object on a If-Modified-Since type query if it hasn't been modified. It should have responded with a 304 response as it did in the TCP_REFRESH_HIT case. Regards Henrik
Re: [squid-users] Objects Release from Cache Earlier Than Expected
I am talking about If-Modified-Since between Squid and the web server, not browser-squid. On ons, 2008-10-22 at 17:57 -0700, BUI18 wrote: Henrik - Thanks for taking time out to respond to my questions. I'm completely stumped on this one. In our production environment, we set min and max to 5 and 7 days, respectively. As I understand it, if the request is made for the object in say3 days or 4 days (less than 5 days), I would always expect a TCP_HIT. But again, after 1 to 2 days, I see TCP_REFRESH_MISS and I get the whole object. I thought that by setting the min to 5 days would guarantee freshness up to 5 days. Do you know of a problem that maybe causes Squid to ignore the rules on determining whether an object is fresh? We used fiddler and actually removed the If-Modified-Since part of the request and still we get TCP_REFRESH_MISS. Do you have any other ideas on areas we might want to check to see what could possibly be causing this behavior? Thanks - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Wednesday, October 22, 2008 4:06:33 PM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On ons, 2008-10-22 at 14:35 -0700, BUI18 wrote: Object is initially cached. Max age in squid.conf is set to 1 min. Before 1 min passes, I request the object and Squid returns TCP_HIT. After 1 min, I try to request for object again. Squid returns TCP_REFRESH_HIT, which is what I expect. I leave the entire system untouched. A day or a day and a half later, I ask for the object again and Squid returns TCP_REFRESH_MISS/200. TCP_HIT is a local hit on the Squid cache. Origin server was not asked. TCP_REFRESH_HIT is a cache hit after the origin server was asked if the object is still fresh. TCP_REFREHS_MISS is when the origin server says the object is no longer fresh and returns a new copy on the conditional query sent by the cache. (same query as in TCP_REFRESH_HIT, different response from the web server). What could possibly cause Squid to refetch the entire object again? A better question is why your server responds with the entire object on a If-Modified-Since type query if it hasn't been modified. It should have responded with a 304 response as it did in the TCP_REFRESH_HIT case. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Objects Release from Cache Earlier Than Expected
But why would Squid even issue an If-Modified-Since to origin server if the min value is set to 5 days? Would this object not be seen as fresh and would just be served up by Squid as a TCP_HIT? - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Wednesday, October 22, 2008 7:19:51 PM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected I am talking about If-Modified-Since between Squid and the web server, not browser-squid. On ons, 2008-10-22 at 17:57 -0700, BUI18 wrote: Henrik - Thanks for taking time out to respond to my questions. I'm completely stumped on this one. In our production environment, we set min and max to 5 and 7 days, respectively. As I understand it, if the request is made for the object in say3 days or 4 days (less than 5 days), I would always expect a TCP_HIT. But again, after 1 to 2 days, I see TCP_REFRESH_MISS and I get the whole object. I thought that by setting the min to 5 days would guarantee freshness up to 5 days. Do you know of a problem that maybe causes Squid to ignore the rules on determining whether an object is fresh? We used fiddler and actually removed the If-Modified-Since part of the request and still we get TCP_REFRESH_MISS. Do you have any other ideas on areas we might want to check to see what could possibly be causing this behavior? Thanks - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Wednesday, October 22, 2008 4:06:33 PM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On ons, 2008-10-22 at 14:35 -0700, BUI18 wrote: Object is initially cached. Max age in squid.conf is set to 1 min. Before 1 min passes, I request the object and Squid returns TCP_HIT. After 1 min, I try to request for object again. Squid returns TCP_REFRESH_HIT, which is what I expect. I leave the entire system untouched. A day or a day and a half later, I ask for the object again and Squid returns TCP_REFRESH_MISS/200. TCP_HIT is a local hit on the Squid cache. Origin server was not asked. TCP_REFRESH_HIT is a cache hit after the origin server was asked if the object is still fresh. TCP_REFREHS_MISS is when the origin server says the object is no longer fresh and returns a new copy on the conditional query sent by the cache. (same query as in TCP_REFRESH_HIT, different response from the web server). What could possibly cause Squid to refetch the entire object again? A better question is why your server responds with the entire object on a If-Modified-Since type query if it hasn't been modified. It should have responded with a 304 response as it did in the TCP_REFRESH_HIT case. Regards Henrik
Re: [squid-users] Objects Release from Cache Earlier Than Expected
On mån, 2008-10-20 at 17:45 -0700, BUI18 wrote: I not sure what you mean by a newer copy of the same URL? Can you elaborate on that a bit? The cache (i.e. Squid) performed a conditional request to the origin web server, and the web server returned a new 200 OK object with full content instead of a small 304 Not Modified. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Objects Release from Cache Earlier Than Expected
The web server is IIS 6. 1) Would there be any reason why it would return the full object when in fact the object has not been modified? 2) If the min age guarantees freshness of the object, why would Squid actually issue and IMS request to the web server in the first place? As I understand it, Squid should only issue and IMS request when objects become STALE. As such, I would have expected Squid to return TCP_HIT instead of TCP_REFRESH_MISS. 3) My big concern is that the store.log shows that the object was released (deleted) from cache well before the min age while there is still and abundant amount of disk space is available. Also, one other question: When Squid issues and IMS request, which date does it use? Is it the date/time that it retrieved the object or is it the Last Modified date/time of the object ascertained by Squid on first retrieval of the object? Regards -bui - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Tuesday, October 21, 2008 3:50:20 AM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On mån, 2008-10-20 at 17:45 -0700, BUI18 wrote: I not sure what you mean by a newer copy of the same URL? Can you elaborate on that a bit? The cache (i.e. Squid) performed a conditional request to the origin web server, and the web server returned a new 200 OK object with full content instead of a small 304 Not Modified. Regards Henrik
Re: [squid-users] Objects Release from Cache Earlier Than Expected
Hi - Here's some additional info I noticed from the store.log. 1224524455.351 SWAPOUT 00 0003A6CB 7377CBD1A7584A5D7C7FD06B5B827595 200 1224524431 1224522501-1 video/jpeg 1337100/1337100 GET http://ftp.mydomain.com/myserver/websites/data/MyVideoFile1020130441180.vid 1224544851.517 RELEASE 00 0003A6CB CD5B96F66CC94483D586D7E67A76A94C 200 1224524431 1224522501-1 video/jpeg 1337100/-279 GET http://ftp.mydomain.com/myserver/websites/data/MyVideoFile1020130441180.vid 1224544862.563 SWAPOUT 00 0003CA26 7377CBD1A7584A5D7C7FD06B5B827595 200 1224544840 1224522501-1 video/jpeg 1337100/1337100 GET http://ftp.mydomain.com/myserver/websites/data/MyVideoFile1020130441180.vid Trace breaks down as follows. 1) File was first pre-fetched by wget program. 2) File was released by cache. 3) File was re-fetch when a user tried to access it. The thing that stuck out is that in the RELEASE line of the log, the real-length is a negative number (-279). What does this mean exactly? - Original Message From: BUI18 [EMAIL PROTECTED] To: squid-users@squid-cache.org Sent: Monday, October 20, 2008 4:02:52 PM Subject: [squid-users] Objects Release from Cache Earlier Than Expected Hi - I have been trying to track down an issue with Squid 2.6 STABLE18 and why users were getting TCP_REFRESH_MISS instead of TCP_REFRESH_HIT on files that were recently cached. We first noticed that users were getting misses when we expected them to receive hits. I have set the min and max age to be 5 and 7 days respectively. When I look in the store.log file, I do see objects which were known to have been cached today (base on time/date stamp in the file name), yet they have status code of RELEASE. The cache_dir (1 TB) on this system is only 25% full. The low watermark is set at 95% with high at 97%. Does any one have any ideas on why Squid would appear to purge the object earlier than expected? Thanks in advance. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: [squid-users] Objects Release from Cache Earlier Than Expected
On mån, 2008-10-20 at 16:02 -0700, BUI18 wrote: Hi - I have been trying to track down an issue with Squid 2.6 STABLE18 and why users were getting TCP_REFRESH_MISS instead of TCP_REFRESH_HIT on files that were recently cached. We first noticed that users were getting misses when we expected them to receive hits. TCP_REFRESH_MISS is a cache validation which indicated the object has been updated on the origin server. I have set the min and max age to be 5 and 7 days respectively. When I look in the store.log file, I do see objects which were known to have been cached today (base on time/date stamp in the file name), yet they have status code of RELEASE. And you are sure it wasn't simply replaced with a newer copy of the same URL? Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Objects Release from Cache Earlier Than Expected
I not sure what you mean by a newer copy of the same URL? Can you elaborate on that a bit? As far as I know, the aspx pages displays a list of buttons for each video file. When the user clicks on the button, it references the URL. I've seen it where the user click the link and gets a TCP_REFRESH_HIT, but if I come back a day later (well within my min/max settings), I get a TCP_REFRESH_MISS. I also previously posted additional info from the store.log. Which shows the object being cached and then released after a short time. - Original Message From: Henrik Nordstrom [EMAIL PROTECTED] To: BUI18 [EMAIL PROTECTED] Cc: squid-users@squid-cache.org Sent: Monday, October 20, 2008 4:55:41 PM Subject: Re: [squid-users] Objects Release from Cache Earlier Than Expected On mån, 2008-10-20 at 16:02 -0700, BUI18 wrote: Hi - I have been trying to track down an issue with Squid 2.6 STABLE18 and why users were getting TCP_REFRESH_MISS instead of TCP_REFRESH_HIT on files that were recently cached. We first noticed that users were getting misses when we expected them to receive hits. TCP_REFRESH_MISS is a cache validation which indicated the object has been updated on the origin server. I have set the min and max age to be 5 and 7 days respectively. When I look in the store.log file, I do see objects which were known to have been cached today (base on time/date stamp in the file name), yet they have status code of RELEASE. And you are sure it wasn't simply replaced with a newer copy of the same URL? Regards Henrik __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
