Re: mod_cache caching the 301 Moved Permanently
At 21/04/05 10:04 (), Devendra Singh wrote: Hi, I am writing to the Developer List because I did not get any response on the Users List and thought that the topic might be relevant to the dev list. If a request comes for a directory w/o trailing slash, it gets cached and the subsequent requests see: Moved Permanently The document has moved here for try to access the URL: http://www.beach-clothing.com/where-to-buy The configuration has two apache 2.0.53. The one running on port 80 has mod_disk_cache which sends requests to 8080 via ProxyPassReverse. Here is my Virtual Host Config: Apache on Port 80: ServerAlias beach-clothing.com www.beach-clothing.com ServerName www.beach-clothing.com ServerAdmin [EMAIL PROTECTED] DocumentRoot /var/www/public_html/beach-clothing ServerPath /beach-clothing RewriteEngine On RewriteCond %{HTTP_HOST} !^www\.beach-clothing\.com [NC] RewriteCond %{HTTP_HOST} !^$ RewriteRule ^/(.*) http://www.beach-clothing.com/$1 [L,R=301] #Enable Caching CacheDisable /index.html CacheEnable disk / RewriteCond %{SERVER_PORT} ^25$ RewriteRule /* - [F] RewriteRule ^(.*)\/$ $1/index.html [P] RewriteRule \.(gif|jpg|png|txt|css|js|ico|swf)$ - [last] RewriteRule ^/(.*)$ http://www.beach-clothing.com:8080/$1 [proxy] ProxyPassReverse / http://www.beach-clothing.com:8080/ Apache on port 8080: ServerAlias beach-clothing.com www.beach-clothing.com ServerName www.beach-clothing.com ServerAdmin [EMAIL PROTECTED] DocumentRoot /var/www/public_html/beach-clothing #Enable Caching for 48 Hours ExpiresActive off ExpiresActive off ExpiresActive on ExpiresByType text/html A432000 UseCanonicalName is off into the Server Config Section of both the Apache. How do I solve this problem? If anyone has any clue / pointer? Hi, Sorry, replying my own post. I modified the "modules/experimental/mod_cache.c" (around line 374) to eliminate the caching of HTTP_MOVED_PERMANENTLY and the problem was solved. /* * what responses should we not cache? * * At this point we decide based on the response headers whether it * is appropriate _NOT_ to cache the data from the server. There are * a whole lot of conditions that prevent us from caching this data. * They are tested here one by one to be clear and unambiguous. */ if (r->status != HTTP_OK && r->status != HTTP_NON_AUTHORITATIVE && r->status != HTTP_MULTIPLE_CHOICES && r->status != HTTP_MOVED_PERMANENTLY --> Remove this line. && r->status != HTTP_NOT_MODIFIED) { reason = apr_psprintf(p, "Response status %d", r->status); } Please comment, is this a right approach? Thanks. Devendra Singh
Re: mod_cache caching the 301 Moved Permanently
On Thu, Apr 21, 2005 at 10:04:54AM +0530, Devendra Singh wrote: > Hi, > > I am writing to the Developer List because I did not get any response on > the Users List and thought that the topic might be relevant to the dev list. > > If a request comes for a directory w/o trailing slash, it gets cached and > the subsequent requests see: > > Moved Permanently > The document has moved here > > for try to access the URL: > http://www.beach-clothing.com/where-to-buy I don't get it. What's your problem? -- justin
Re: mod_cache caching the 301 Moved Permanently
On 4/22/05, Justin Erenkrantz <[EMAIL PROTECTED]> wrote: > On Thu, Apr 21, 2005 at 10:04:54AM +0530, Devendra Singh wrote: > > Hi, > > > > I am writing to the Developer List because I did not get any response on > > the Users List and thought that the topic might be relevant to the dev list. > > > > If a request comes for a directory w/o trailing slash, it gets cached and > > the subsequent requests see: > > > > Moved Permanently > > The document has moved here > > > > for try to access the URL: > > http://www.beach-clothing.com/where-to-buy > > I don't get it. What's your problem? -- justin The 'here' link is to http://www.beach-clothing.com:8080/where-to-buy/ while he wants it do be to http://www.beach-clothing.com/where-to-buy/
Re: mod_cache caching the 301 Moved Permanently
Sander Striker wrote: > [EMAIL PROTECTED] wrote: > >> The problem seems to be, that the proxied backend server that is >> cached via mod_disk_cache originally >> delivers HTTP status 301 and the Location >> http://www.beach-clothing.com/where-to-buy/, but once cached >> mod_disk_cache delivers HTTP status 200 instead of 301 (but correctly >> redelivering the Location header). >> I have not proved this for myself so far, but this seems the problem >> to me. > > > This wouldn't surprise me one bit. The 2.1 branch has seen quite a bit > of churn in this > area. > > Any chance you could give 2.1 a go and see if that works correctly? Sorry not enough time right now to check this on the living object. But I think what is missing is something like the following (untested) patch (done against 2.0.54, but code in trunk is the same at this part): --- mod_cache.c.orig2005-04-11 17:47:03.0 +0200 +++ mod_cache.c 2005-04-22 23:43:19.0 +0200 @@ -220,6 +220,8 @@ static int cache_out_filter(ap_filter_t ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server, "cache: running CACHE_OUT filter"); +/* restore status of cached response */ +r->status = cache->handle->cache_obj->info.status; /* recall_headers() was called in cache_select_url() */ cache->provider->recall_body(cache->handle, r->pool, bb); I found no place in the code where the saved status code is restored to the response, so I think this could be the right place to do so. Could somebody crosscheck who is more familar with the code? Sander? Justin? > > Sander Regards Rüdiger
Re: mod_cache caching the 301 Moved Permanently
Olaf van der Spek wrote: > On 4/22/05, Justin Erenkrantz <[EMAIL PROTECTED]> wrote: > [..cut..] >> >>I don't get it. What's your problem? -- justin > > > The 'here' link is to http://www.beach-clothing.com:8080/where-to-buy/ > while he wants it do be to http://www.beach-clothing.com/where-to-buy/ > > His problem is a different one. If you type http://www.beach-clothing.com/where-to-buy (without the slash) you have the following headers (first my request / second server reply): GET /where-to-buy HTTP/1.1 Host: www.beach-clothing.com User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050319 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai n;q=0.8,image/png,*/*;q=0.5 Accept-Language: de,en;q=0.8,de-de;q=0.5,en-gb;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive HTTP/1.x 200 OK Date: Fri, 22 Apr 2005 20:39:58 GMT Server: Apache Content-Type: text/html; charset=iso-8859-1 Location: http://www.beach-clothing.com/where-to-buy/ Cache-Control: max-age=432000 Expires: Tue, 26 Apr 2005 04:58:38 GMT Content-Length: 256 Age: 142880 Keep-Alive: timeout=3, max=20 Connection: Keep-Alive The problem seems to be, that the proxied backend server that is cached via mod_disk_cache originally delivers HTTP status 301 and the Location http://www.beach-clothing.com/where-to-buy/, but once cached mod_disk_cache delivers HTTP status 200 instead of 301 (but correctly redelivering the Location header). I have not proved this for myself so far, but this seems the problem to me. I think the difference between the here link and the Location header can be explained by the ProxyPassReverse directive which rewrites the Location header, but not the HTML code in the error page. Regards Rüdiger
Re: mod_cache caching the 301 Moved Permanently
[EMAIL PROTECTED] wrote: The problem seems to be, that the proxied backend server that is cached via mod_disk_cache originally delivers HTTP status 301 and the Location http://www.beach-clothing.com/where-to-buy/, but once cached mod_disk_cache delivers HTTP status 200 instead of 301 (but correctly redelivering the Location header). I have not proved this for myself so far, but this seems the problem to me. This wouldn't surprise me one bit. The 2.1 branch has seen quite a bit of churn in this area. Any chance you could give 2.1 a go and see if that works correctly? Sander
Re: mod_cache caching the 301 Moved Permanently
At 22/04/2005 11:21 (), Olaf van der Spek wrote: On 4/22/05, Justin Erenkrantz <[EMAIL PROTECTED]> wrote: > On Thu, Apr 21, 2005 at 10:04:54AM +0530, Devendra Singh wrote: > > Hi, > > > > I am writing to the Developer List because I did not get any response on > > the Users List and thought that the topic might be relevant to the dev list. > > > > If a request comes for a directory w/o trailing slash, it gets cached and > > the subsequent requests see: > > > > Moved Permanently > > The document has moved here > > > > for try to access the URL: > > http://www.beach-clothing.com/where-to-buy > > I don't get it. What's your problem? -- justin The 'here' link is to http://www.beach-clothing.com:8080/where-to-buy/ while he wants it do be to http://www.beach-clothing.com/where-to-buy/ When the Front-End Apache (running on port 80) gets a request for http://www.beach-clothing.com/where-to-buy (notice the lack of trailing slash), it issues a 301 Moved Permanently to http://www.beach-clothing.com/where-to-buy/ (notice the added trailing slash). The Front-End Apache proxies this request to Back-End Apache running on 8080 which becomes http://www.beach-clothing.com:8080/where-to-buy/ and this is Cached by the Front-End Apache. The subsequent request for this URL gives http://www.beach-clothing.com:8080/where-to-buy/ from the Cache. This is not as expected. Thus I feel that either 301 responses should not be proxied or they should not be Cached at all. Thanks. Devendra Singh
Re: mod_cache caching the 301 Moved Permanently
Hi, it has been some time since the original thread. This is in reply to [1]. Sander Striker wrote: >> [EMAIL PROTECTED] wrote: >> The problem seems to be, that the proxied backend server that is >> cached via mod_disk_cache originally >> delivers HTTP status 301 and the Location >> http://www.beach-clothing.com/where-to-buy/, but once cached >> mod_disk_cache delivers HTTP status 200 instead of 301 (but >> correctly redelivering the Location header). >> I have not proved this for myself so far, but this seems the problem >> to me. >This wouldn't surprise me one bit. The 2.1 branch has seen quite a >bit of churn in this >area. >Any chance you could give 2.1 a go and see if that works correctly? It is the same problem as described in issue 32226 [2]. Unfortunately, the described behavior is still present in both httpd 2.0.54 and 2.1.6-alpha with mod_disk_cache active. I can not reproduce it with 2.1.6-alpha and mod_mem_cache. Tested with the following setup: Frontend: httpd 2.1.6-alpha on 127.0.0.1, cache, disk_cache and proxy* modules loaded, out of the box httpd.conf plus: CacheRoot /cache/alpha CacheEnable disk / CacheDirLevels 1 CacheDirLength 2 ProxyPass/ http://backend.tld/ ProxyPassReverse / http://backend.tld/ Backend: Apache 1.3, a file cachetest/gonk/index.html relative to DocumentRoot. A first request to the Proxy (GET /cachetest/gonk) correctly responds in 301ing to /cachetest/gonk/ first -- and then getting the index document. There are 2 .data/.header couples in the cachedir at this point (the redirect-message and the index document ). A second request with a second client, or if the browser's cache has been cleared, returns the cached "Moved Permanently" message with status 200. This message contains a link in the backend's view of the URLs (as discussed in the original thread). The /cache/alpha/Kx/[EMAIL PROTECTED] Cache File contains: [EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@@-a<89>Nû^C^@@Žô<8C>[EMAIL PROTECTED]<89>[EMAIL PROTECTED]<89>[EMAIL PROTECTED]/cachetest/gonk?Date: Thu, 07 Jul 2005 12:34 :21 GMT Server: Apache Location: http://127.0.0.1/cachetest/gonk/ Content-Type: text/html; charset=iso-8859-1 Expires: Thu, 07 Jul 2005 12:35:21 GMT Host: 127.0.0.1 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050513 Debian/1.7.8-1 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Accept-Language: de-at,de;q=0.8,en;q=0.5,it;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Max-Forwards: 10 X-Forwarded-For: 127.0.0.1 X-Forwarded-Host: 127.0.0.1 X-Forwarded-Server: 127.0.0.1 [3], which is an attachment to [2] contains a fragment that keeps mod_cache.c from handling redirects at all -- if i understood it right, however it works for us. This is probably only the second best i approach, since 301 is cacheable. Regards, Hansjörg References: [1] http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/[EMAIL PROTECTED] [2] http://issues.apache.org/bugzilla/show_bug.cgi?id=32226 [3] http://issues.apache.org/bugzilla/attachment.cgi?id=13433 -- IT ServicesUniversity of Innsbruck CFB4 D6E7 33F4 34C0 18B9 6661 E355 4337 3F8B D9C2 http://purl.org/net/hansjoerg.pehofer/public_key signature.asc Description: Digital signature
Re: mod_cache caching the 301 Moved Permanently
Have you checked http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/[EMAIL PROTECTED] ? It contains a small patch which was not discussed any further here. Regards Rüdiger Hansjoerg Pehofer wrote: > Hi, > > it has been some time since the original thread. > This is in reply to [1].
Re: mod_cache caching the 301 Moved Permanently
On Thu, Jul 07, 2005 at 09:22:25PM +0200, [EMAIL PROTECTED] wrote: > Have you checked > http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/[EMAIL > PROTECTED] ? > > It contains a small patch which was not discussed any further here. Hi, i tried your patch today. Unfortunately cache->handle->cache_obj->info.status isn't set during cache_out_filter(), but cache->handle->status is. (It gets set in mod_disk_cache.c:499 during recall_headers().) I tried to use that and it seems to work. However, not knowing much of how the whole thing fits together, i do not claim that this is the solution. An updated version of your patch below. Thanks && Regards Hansjoerg --- mod_cache.c.orig2005-04-11 17:47:03.0 +0200 +++ mod_cache.c 2005-07-11 16:52:35.913264943 +0200 @@ -219,7 +219,10 @@ ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server, "cache: running CACHE_OUT filter"); - + +/* restore status of cached response */ +r->status = cache->handle->status; + /* recall_headers() was called in cache_select_url() */ cache->provider->recall_body(cache->handle, r->pool, bb); -- IT ServicesUniversity of Innsbruck CFB4 D6E7 33F4 34C0 18B9 6661 E355 4337 3F8B D9C2 http://purl.org/net/hansjoerg.pehofer/public_key signature.asc Description: Digital signature
Re: mod_cache caching the 301 Moved Permanently
Hansjoerg Pehofer wrote: > On Thu, Jul 07, 2005 at 09:22:25PM +0200, [EMAIL PROTECTED] wrote: > >>Have you checked >>http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/[EMAIL >>PROTECTED] ? >> >>It contains a small patch which was not discussed any further here. > > > Hi, > > i tried your patch today. Unfortunately > cache->handle->cache_obj->info.status isn't set during cache_out_filter(), > but cache->handle->status is. > (It gets set in mod_disk_cache.c:499 during recall_headers().) Many thanks for the feedback and the corrected patch. I checked my patch again and must admit that it does only make sense on Apache 2.1. For Apache 2.0.x your patch is the correct one. Question to the mod_cache gurus: While mod_mem_cache restores the status during recall_headers in Apache 2.0.54 it seems that mod_mem_cache does not do this in Apache 2.1.x (Maybe fixed meanwhile my latest revision of mod_mem_cache.c is 178625 and viewcvs is currently down), whereas mod_disk_cache still does this in Apache 2.1.x in file_cache_recall_mydata which is called by open_entity. Is this behaviour intended? Regards Rüdiger [..cut..]
Re: mod_cache caching the 301 Moved Permanently
[EMAIL PROTECTED] wrote: Hansjoerg Pehofer wrote: On Thu, Jul 07, 2005 at 09:22:25PM +0200, [EMAIL PROTECTED] wrote: Have you checked http://mail-archives.apache.org/mod_mbox/httpd-dev/200504.mbox/[EMAIL PROTECTED] ? It contains a small patch which was not discussed any further here. Hi, i tried your patch today. Unfortunately cache->handle->cache_obj->info.status isn't set during cache_out_filter(), but cache->handle->status is. (It gets set in mod_disk_cache.c:499 during recall_headers().) Many thanks for the feedback and the corrected patch. I checked my patch again and must admit that it does only make sense on Apache 2.1. For Apache 2.0.x your patch is the correct one. Question to the mod_cache gurus: While mod_mem_cache restores the status during recall_headers in Apache 2.0.54 it seems that mod_mem_cache does not do this in Apache 2.1.x (Maybe fixed meanwhile my latest revision of mod_mem_cache.c is 178625 and viewcvs is currently down), whereas mod_disk_cache still does this in Apache 2.1.x in file_cache_recall_mydata which is called by open_entity. Is this behaviour intended? 2.0 is quite a bit behind when it comes to mod_cache. mod_mem_cache has gotten a lot less attention in 2.1 as mod_disk_cache has. That said, in 2.1, I think we should restore status in mod_cache, not in the cache providers. Ofcourse, if that isn't possible for whatever reason, the restoration has to be done in _all_ cache providers. The above might explain the discrepancies... Sander
Re: mod_cache caching the 301 Moved Permanently
Sander Striker wrote: > [EMAIL PROTECTED] wrote: [..cut..] >> Question to the mod_cache gurus: >> While mod_mem_cache restores the status during recall_headers in >> Apache 2.0.54 >> it seems that mod_mem_cache does not do this in Apache 2.1.x (Maybe >> fixed meanwhile >> my latest revision of mod_mem_cache.c is 178625 and viewcvs is >> currently down), whereas >> mod_disk_cache still does this in Apache 2.1.x in >> file_cache_recall_mydata which is >> called by open_entity. Is this behaviour intended? > > > 2.0 is quite a bit behind when it comes to mod_cache. mod_mem_cache has > gotten > a lot less attention in 2.1 as mod_disk_cache has. > > That said, in 2.1, I think we should restore status in mod_cache, not in > the cache Sorry for being confused. But maybe my question was not completely correct. mod_disk_cache does not restore r->status directly but it restores the status information to cache->handle->cache_obj->info.status. This gives mod_cache the opportunity to restore r->status from this (as my patch does). The problem is that mod_mem_cache does not seem to restore the status information at all from its storage. This way mod_cache has no chance to restore it in this case. I agree with you that mod_cache should set r->status with the information restored by the provider to cache->handle->cache_obj->info.status. mod_cache should take the final decision if it does that or not. This decision should not be done by the providers. > providers. Ofcourse, if that isn't possible for whatever reason, the > restoration > has to be done in _all_ cache providers. The above might explain the > discrepancies... But all cache providers have to store it in their storage and have to restore it from there to the caching data structure. Maybe this is what you wanted to say? Regards Rüdiger
Re: mod_cache caching the 301 Moved Permanently
[EMAIL PROTECTED] wrote: Sander Striker wrote: [...] That said, in 2.1, I think we should restore status in mod_cache, not in the cache Sorry for being confused. But maybe my question was not completely correct. It was fine; I'm being 'lazy' and not looking at the code, but going by memory instead. Sometimes I forget pieces... mod_disk_cache does not restore r->status directly but it restores the status information to cache->handle->cache_obj->info.status. This gives mod_cache the opportunity to restore r->status from this (as my patch does). The problem is that mod_mem_cache does not seem to restore the status information at all from its storage. Right, this is where mod_disk_cache did get some attention and mod_mem_cache didn't yet. This way mod_cache has no chance to restore it in this case. I agree with you that mod_cache should set r->status with the information restored by the provider to cache->handle->cache_obj->info.status. mod_cache should take the final decision if it does that or not. This decision should not be done by the providers. Indeed. [...] providers. Ofcourse, if that isn't possible for whatever reason, the restoration has to be done in _all_ cache providers. The above might explain the discrepancies... But all cache providers have to store it in their storage and have to restore it from there to the caching data structure. Maybe this is what you wanted to say? Yup. Sander