RE: rewrite processing before cache
Good to know about that trunk feature. I'll have to remember that when it gets around to stable. The vary thing i might try. I'll have to weigh cost versus functionality. Thanx very much. -Tony Sent from my Windows® phone. -Original Message- From: Eric Covener Sent: Monday, March 01, 2010 6:50 PM To: modules-dev@httpd.apache.org Subject: Re: rewrite processing before cache On Mon, Mar 1, 2010 at 8:36 PM, Anthony J. Biacco wrote: > I asked a vaguely similar question about a year ago, but this one is a > little more simplistic in comparison. > > Running 2.2.14 on linux, disk caching a servlet's output, not ignoring > query string as that can change the output, and not using > cacheignoreheaders. > > My problem is, using rewrite rules on such requests don't always get > triggered. > For example, if I'm trying to block a browser through a rewrite rule for > a request to said servlet, sometimes the cache takes over and serves the > cached output instead of following the rewrite. i.e. sometimes the > browser will get blocked and sometimes it will get cached content. > I'm wondering what's the workflow on such a scenario, and how can I > configure it so the rewrite always gets processed before a cache check, > if possible. Trunk has a feature called "CacheQuickHandler" that stops the cache from short-circuiting all the local processing, but not available in 2.2.x. The other option you have is to add a Vary header for user-agent, but that makes you store a ton of different copies in your cache due to the variety in user-agents. -- Eric Covener cove...@gmail.com
rewrite processing before cache
I asked a vaguely similar question about a year ago, but this one is a little more simplistic in comparison. Running 2.2.14 on linux, disk caching a servlet's output, not ignoring query string as that can change the output, and not using cacheignoreheaders. My problem is, using rewrite rules on such requests don't always get triggered. For example, if I'm trying to block a browser through a rewrite rule for a request to said servlet, sometimes the cache takes over and serves the cached output instead of following the rewrite. i.e. sometimes the browser will get blocked and sometimes it will get cached content. I'm wondering what's the workflow on such a scenario, and how can I configure it so the rewrite always gets processed before a cache check, if possible. Thanx, -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com
RE: Preventing the use of chunked TF encoding while content-filtering
Christoph, I had a mod_buffer module written for me by Konstantin Chuguev (konstan...@chuguev.com) which collects chunks and buffers them for transfer in one shot. You should contact him and see whether he'll give/license it to you. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Christoph Gröver [mailto:gro...@sitepark.com] > Sent: Monday, November 09, 2009 11:07 AM > To: modules-dev@httpd.apache.org > Subject: Preventing the use of chunked TF encoding while content- > filtering > > > Hello list, > > > I have written a module which does filtering the content. > > It gets those buckets and works on them and passes on the brigade of > buckets. OK, works perfect for nearly all conditions. > > Execpt > > When the user runs the infamous Internet Explorer and uses a > misconfigured proxy, it doesn't. > > With the help of some debugging we have found out what happens: The > proxy is not HTTP/1.1-aware, which means it just changes the > HTTP-response to be HTTP/1.0, but it keeps the > 'chunked transfer encoding' of the original content. > > Almost all browsers recognize that it's still chunked although the > header claims to be HTTP/1.0 and do the right thing. > > MSIE doesn't. It gives the user the content and displays it with those > hex encoded chunk lengths in it. > > Of course, this breaks website design and often the functionality. > > So, what can we do about it? > > We cannot change the proxy people are using and we cannot fix their > misconfigured proxies (or their broken proxies). > > We cannot tell them to not use MSIE, either. > > > So we should tell the Apache Webserver to not use 'chunked transfer > encoding'. I thought this might be possible by just saying > > r->chunked = FALSE; > > But it didn't help. > > So after some talking: Is there a way to get rid of CTFE ? > > Perhaps if we collect all the chunks, put it in one chunk and set a > ContentLength-Header ? > > Or is there another trick to do this? > > Greetings from Münster, looking forward to your ideas. > > -- > Christoph Gröver, gro...@sitepark.com > Sitepark GmbH, Gesellschaft für Informationsmanagement, AG Münster, HRB > 5017 Rothenburg 14-16, D-48143 Münster, Telefon (0251) 48265-50 > Geschäftsführer: Dipl.-Phys. Martin Kurze, Dipl.-Des. Thorsten Liebold
RE: mod_deflate feature needed
Yep, I think it got lost nick. I'd go along with the filter idea after deflate if it does what I need. Are you referring to mod_filter based directives such as FilterProvider/FilterProtocol? The client/partner/vendor isn't requiring it to support HTTP, but to support their services. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Nick Kew [mailto:n...@apache.org] > Sent: Thursday, July 16, 2009 10:46 AM > To: modules-dev@httpd.apache.org > Subject: Fwd: mod_deflate feature needed > > Looks like this got lost in the ether ... > > Begin forwarded message: > > > > > > On 15 Jul 2009, at 23:39, Anthony J. Biacco wrote: > > > >> I'm trying to use mod_deflate to compress data coming out of tomcat > >> through mod_jk and need the proper content-length header set for the > >> COMPRESSED data, but can't do this because the data is streamed > >> and sent > >> after the headers are set, therefore we don't know the compressed > >> content-length until after the fact. > >> I'd either like to request a option to enable such a feature where > >> I can > >> have the compressed data buffered, the headers set, and then the > data > >> sent. > > > > That's the wrong approach. Think modular! > > > > The right approach is to insert another filter after mod_deflate > > to do your buffering (and of course note its effect on performance > > and potential role in a DoS attack). The existing content_length > > filter would make a startingpoint. > > > > Or, better, fix your client to support HTTP, without the need for > > a Content-Length header. > > > > -- > > Nick Kew
RE: mod_deflate feature needed
Yep, that's what I said, should. It doesn't require it and I understand why I's chunking and streaming it for speed of sending. But why should it send the content-length out properly for static files and not use chunked encoding, and fail to do the same for my tomcat content and use chunked encoding (especially when I'm not setting chunked encoding to start with on the data)? That's not consistent. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Tom Evans [mailto:tevans...@googlemail.com] > Sent: Thursday, July 16, 2009 10:52 AM > To: modules-dev@httpd.apache.org > Subject: RE: mod_deflate feature needed > > On Thu, 2009-07-16 at 10:20 -0600, Anthony J. Biacco wrote: > > We're not violating the specs, we're following them. The HTTP spec > > states that you should send out a content-length header with the > correct > > trasnfer length, and mod_deflate in this case is not doing that. > > > > -Tony > > No it doesn't; it says that 'Applications SHOULD use this field to > indicate the transfer-length of the message-body, unless this is > prohibited by the rules in section 4.4' > > Rule 2 of section 4.4 clearly indicates that you must not set it if you > are transferring data using the chunked transfer encoding (in fact, any > transfer encoding other than 'identity'). > > It is sub optimal in most use cases to buffer the entire response and > gzip encode it, so mod_deflate does the sane thing and so transfers it > in chunks. > > Cheers > > Tom
RE: mod_deflate feature needed
Sorry, CDN is Content Delivery Network. Akamai, Limelight, etc.. While the CDN supports chunked transfer fine for actually delivering data, to utilize its caching services (which is the whole point of the CDN obviously) it wants the content-length header from my servers. It's stupid, yes, but getting them to change their software to not require this seems very unlikely. The spec seems to state transfer length, not content length. To me that means the size of the actual encoded stream, which would be the compressed size in a gzip state or the uncompressed size in the non-gzip state. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Houser, Rick [mailto:houser.r...@aoins.com] > Sent: Thursday, July 16, 2009 10:48 AM > To: modules-dev@httpd.apache.org > Subject: RE: mod_deflate feature needed > > I'm not familiar with the CDN acronym. I could be mistaken, but I'm > fairly certain that HTTP 1.1 required support for chunked transfer > (which implies Content-Length would NOT be required for anything HTTP > 1.1 compliant). Additionally, I thought the content-length referenced > the contents of the file without taking the content-encoding into > account. > > > > Thanks, > > Rick Houser > Auto-Owners Insurance > Systems Support > (517)703-2580 > > -Original Message- > From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] > Sent: Thursday, July 16, 2009 12:42 PM > To: modules-dev@httpd.apache.org > Subject: RE: mod_deflate feature needed > > I can get the compressed content, that's not a problem. My thing is, > I'm > trying to cache said response at my CDN. The CDN requires a > content-length header (among other things) to do this. And with the > chunked encoding I'm not getting one (fyi, I'm not doing chunked from > tomcat, I'm actually sending the content-length). > > -Tony > --- > Manager, IT Operations > Format Dynamics, Inc. > 303-573-1800x27 > abia...@formatdynamics.com > http://www.formatdynamics.com > > > > -Original Message- > > From: Sorin Manolache [mailto:sor...@gmail.com] > > Sent: Thursday, July 16, 2009 10:37 AM > > To: modules-dev@httpd.apache.org > > Subject: Re: mod_deflate feature needed > > > > On Thu, Jul 16, 2009 at 00:39, Anthony J. > > Biacco wrote: > > > I'm trying to use mod_deflate to compress data coming out of tomcat > > > through mod_jk and need the proper content-length header set for > the > > > > COMPRESSED data, but can't do this because the data is streamed and > > sent > > > after the headers are set, therefore we don't know the compressed > > > content-length until after the fact. > > > I'd either like to request a option to enable such a feature where > I > > can > > > have the compressed data buffered, the headers set, and then the > data > > > sent. > > > I'd be willing to pay someone here a nominal free to do it > privately, > > if > > > one so wishes to do it. > > > > Have you tried placing the JkMount directive inside a Location and > > using just one argument for JkMount? > > > > > > JkMount ajp13_worker > > SetOutputFilter DEFLATE > > > > > > In my case it works, I get a compressed response. > > > > Here are the relevant response headers: > > > > HTTP/1.1 200 OK > > Date: Thu, 16 Jul 2009 16:28:19 GMT > > Server: Apache/2.2.11 (Debian) mod_jk/1.2.26 mod_perl/2.0.4 > > Perl/v5.10.0 > > Accept-Ranges: bytes > > ETag: W/"1127-1245356448000" > > Last-Modified: Thu, 18 Jun 2009 20:20:48 GMT > > Vary: Accept-Encoding > > Content-Encoding: gzip > > Transfer-Encoding: chunked > > Content-Type: text/html > > > > As the transfer is chunked, there's no Content-Length but the > response > > > is still compressed. > > > > Are you constrained not to use chunked transfer? > > > > Sorin > > > > > > -- > > A: Because it reverses the logical flow of conversation. > > Q: Why is top-posting frowned upon? > > A: Top-posting. > > Q: What is the most annoying thing in e-mail? >
RE: mod_deflate feature needed
I can get the compressed content, that's not a problem. My thing is, I'm trying to cache said response at my CDN. The CDN requires a content-length header (among other things) to do this. And with the chunked encoding I'm not getting one (fyi, I'm not doing chunked from tomcat, I'm actually sending the content-length). -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Sorin Manolache [mailto:sor...@gmail.com] > Sent: Thursday, July 16, 2009 10:37 AM > To: modules-dev@httpd.apache.org > Subject: Re: mod_deflate feature needed > > On Thu, Jul 16, 2009 at 00:39, Anthony J. > Biacco wrote: > > I'm trying to use mod_deflate to compress data coming out of tomcat > > through mod_jk and need the proper content-length header set for the > > COMPRESSED data, but can't do this because the data is streamed and > sent > > after the headers are set, therefore we don't know the compressed > > content-length until after the fact. > > I'd either like to request a option to enable such a feature where I > can > > have the compressed data buffered, the headers set, and then the data > > sent. > > I'd be willing to pay someone here a nominal free to do it privately, > if > > one so wishes to do it. > > Have you tried placing the JkMount directive inside a Location and > using just one argument for JkMount? > > > JkMount ajp13_worker > SetOutputFilter DEFLATE > > > In my case it works, I get a compressed response. > > Here are the relevant response headers: > > HTTP/1.1 200 OK > Date: Thu, 16 Jul 2009 16:28:19 GMT > Server: Apache/2.2.11 (Debian) mod_jk/1.2.26 mod_perl/2.0.4 > Perl/v5.10.0 > Accept-Ranges: bytes > ETag: W/"1127-1245356448000" > Last-Modified: Thu, 18 Jun 2009 20:20:48 GMT > Vary: Accept-Encoding > Content-Encoding: gzip > Transfer-Encoding: chunked > Content-Type: text/html > > As the transfer is chunked, there's no Content-Length but the response > is still compressed. > > Are you constrained not to use chunked transfer? > > Sorin > > > -- > A: Because it reverses the logical flow of conversation. > Q: Why is top-posting frowned upon? > A: Top-posting. > Q: What is the most annoying thing in e-mail?
RE: mod_deflate feature needed
We're not violating the specs, we're following them. The HTTP spec states that you should send out a content-length header with the correct trasnfer length, and mod_deflate in this case is not doing that. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com > -Original Message- > From: Houser, Rick [mailto:houser.r...@aoins.com] > Sent: Thursday, July 16, 2009 7:01 AM > To: modules-dev@httpd.apache.org > Subject: RE: mod_deflate feature needed > > I don't understand why standard mod_deflate functionality won't work. > If you violate the specs and muck with content-length, it's not going > to > work right in a browser, anyhow. > > > > Thanks, > > Rick Houser > Auto-Owners Insurance > Systems Support > (517)703-2580 > > -Original Message- > From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] > Sent: Wednesday, July 15, 2009 6:40 PM > To: modules-dev@httpd.apache.org > Subject: mod_deflate feature needed > > I'm trying to use mod_deflate to compress data coming out of tomcat > through mod_jk and need the proper content-length header set for the > COMPRESSED data, but can't do this because the data is streamed and > sent > after the headers are set, therefore we don't know the compressed > content-length until after the fact. > I'd either like to request a option to enable such a feature where I > can > have the compressed data buffered, the headers set, and then the data > sent. > I'd be willing to pay someone here a nominal free to do it privately, > if > one so wishes to do it. > > Thanx, > > -Tony > --- > Manager, IT Operations > Format Dynamics, Inc. > 303-573-1800x27 > abia...@formatdynamics.com > http://www.formatdynamics.com > > >
mod_deflate feature needed
I'm trying to use mod_deflate to compress data coming out of tomcat through mod_jk and need the proper content-length header set for the COMPRESSED data, but can't do this because the data is streamed and sent after the headers are set, therefore we don't know the compressed content-length until after the fact. I'd either like to request a option to enable such a feature where I can have the compressed data buffered, the headers set, and then the data sent. I'd be willing to pay someone here a nominal free to do it privately, if one so wishes to do it. Thanx, -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com
RE: rewrite before caching?
Yeah, it would create twice the number of requests to my server. One for the initial hit and the other for the redirect. If it was a small % of my hits, I wouldn't care, but this url is about 90% of my web server hits. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Friday, March 20, 2009 6:51 AM To: modules-dev@httpd.apache.org Subject: RE: rewrite before caching? Is there a reason you can't just change your rewrite rule to a [R=301]? Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message----- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Thursday, March 19, 2009 4:53 PM To: modules-dev@httpd.apache.org Subject: RE: rewrite before caching? Sorry, no, these are in httpd.conf, I don't use any htaccess files. The situation is, this particular path was going to tomcat through mod_jk, and initially the dev guys didn't want the data cached in tomcat, so they introduced the killCache parameter in referencing content (before my existence in the company). But then down the road I decided to start caching this data in apache to relieve unnecessary load on our tomcats (and with the understanding that at this time in our application's timeline, the data now could be cached). And while we changed the referencing urls that we had access to, to remove the killCache parameter, there are some references out of our control (currently) that still use it. So I want to now take those requests and implement the caching for them also until the point where we can get the people controlling the remaining references to change them. On the current (changed) url, I'm now setting the appropriate cache-control and expire headers. My current idea is using rewrite I can set an environmental variable on the existence of killCache in the query string and then use that to set a no-cache header and unset the expires header, that will make sure the killCache request doesn't get cached to disk. The internally rewritten url won't get served from cache then also, but I figure I can always do an internal proxy with the [P] flag, and then the proxied rewritten request will pull from cache. I'll still get 2 requests in the log due to the proxy request, but I can use a SetEnvIf Remote_Host to check for the local server ip (denoting the proxied request) and not log the entry based on that. It's ugly, I hate that I have to do it, and the proxy request will still consume more resources than I'd like, but may just be more beneficial than sending the killCache requests through to tomcat. Opinions would be appreciated on my suggestion, privately if you want, as I don't want to get too OT here. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Ray Morris [mailto:supp...@bettercgi.com] Sent: Thursday, March 19, 2009 2:22 PM To: modules-dev@httpd.apache.org Subject: Re: rewrite before caching? If you're doing your rewrite in .htaccess, it _MAY_ help to move that rewrite to httpd.conf. In many cases rewrite rules in httpd.conf get processed at an earlier stage than those in .htaccess. This is becase Apace can't read the .htacces file until after it resolves the URL into a filename, so it knows which directories to look for .htaccess files in. That occurs quite late in the process, during the fixup hook. Other than that, I can only suggest that there are several http headers designed for the purpose of controlling caching so that just the right things are cached for just the right amount of time, and only under the right conditions. Using that carefully designed and well established system may yield better results than the "big hammer" approach of sticking bogus junk into the URL, then trying to remove it later for certain kinds of caching. You've added that to try to prevent caching, yet clearly the content SHOULD be cached in some cases, as is clear because you are actually wanting to cache it. Rather than inventing your own caching system, look into setting the appropriate headers so that it's cached when appropriate and for the appropriate amount of time. -- Ray B. Morris supp...@bettercgi.com Strongbox - The next generation in site security: http://www.bettercgi.com/strongbox/ Throttlebox - Intelligent Bandwidth Control http://www.bettercgi.com/throttlebox/ Strongbox / Throttlebox affiliate program: http://www.bettercgi.com/affiliates/user/register.php On 03/19/2009 02:36:23 PM, Anthony J. Biacco wrote: > I posted this on the users list, but didn't get any help, so I'm > hoping the dev people here can either help or at least explain if what
rewrite before caching?
I posted this on the users list, but didn't get any help, so I'm hoping
the dev people here can either help or at least explain if what I'm
seeing below is how it's meant to work..
I have a uri with cachebusting in it that looks like this:
/path?killCache=x¶meter1=y%parameter2=z
where x is a unix timestamp used to make sure the url isn't cached by
the browser.
Now though I want to use apache disk caching on this url. Obviously with
the killCache parameters always changing though, this is futile. And I
can't ignore the query string with CacheIgnoreQueryString because
parameter1 and parameter2 are important to the output to the end-user,
so the cache would be inconsistent if I had caching ignore the query
string.
So I decided to rewrite the url to remove the killCache parameter, and
while this rule itself works fine (I'm rewriting to a path not a full
url), it seems the cache check is done before the rewrite.
My rule is:
RewriteCond %{QUERY_STRING}
^killCache=(.+)\¶meter1=(.+)\¶meter2=(.+)
RewriteRule ^/path /path?parameter1=%2¶meter2=%3 [L]
And in the log I see:
[Wed Mar 18 12:02:47 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE
filter for /path
[Wed Mar 18 12:02:47 2009] [debug] mod_cache.c(138): Adding
CACHE_REMOVE_URL filter for /path
[Wed Mar 18 12:02:47 2009] [debug] mod_headers.c(740): headers:
ap_headers_output_filter()
mod_cache.c(639): cache: Caching url:
/path?killCache=x¶meter1=y¶meter2=z
[Wed Mar 18 12:02:47 2009] [debug] mod_cache.c(645): cache: Removing
CACHE_REMOVE_URL filter.
[Wed Mar 18 12:02:47 2009] [debug] mod_disk_cache.c(962): disk_cache:
Stored headers for URL
http://myhost/path?killCache=x¶meter1=y¶meter2=z
[Wed Mar 18 12:02:47 2009] [debug] mod_disk_cache.c(1051): disk_cache:
Body for URL http://myhost/path?killCache=x¶meter1=y¶meter2=z
cached.
And I get a different cache file stored for every value of the parameter
killCache.
Anybody know a way to get the rewriterule processed before the cache
mechanism? I can rewrite to a full url and the new request will then
pull from cache, but that induces an extra request and an extra log
entry, which I'd like to avoid if at all possible.
Thanx,
-Tony
---
Manager, IT Operations
Format Dynamics, Inc.
303-573-1800x27
abia...@formatdynamics.com
http://www.formatdynamics.com
RE: mem_cache and disk_cache acting differently when determine when to cache content?
I have 5 children, so Eric's answer that each process has its own in-memory cache makes sense. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Monday, February 23, 2009 6:22 AM To: modules-dev@httpd.apache.org; us...@httpd.apache.org Subject: RE: mem_cache and disk_cache acting differently when determine when to cache content? Is this the only child process, or do you have 3+? Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Friday, February 20, 2009 6:12 PM To: us...@httpd.apache.org Cc: modules-dev@httpd.apache.org Subject: RE: mem_cache and disk_cache acting differently when determine when to cache content? I stand corrected, The mem_cache module DOES serve out the cached version, but not until the 6th request of the URL. The disk_cache module serves out the cached version on the 2nd request (as it should). FYI, I'm using the worker mpm. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message----- From: Anthony J. Biacco Sent: Friday, February 20, 2009 4:05 PM To: 'us...@httpd.apache.org' Cc: 'modules-dev@httpd.apache.org' Subject: mem_cache and disk_cache acting differently when determine when to cache content? Hi, I'm serving out dynamic content from apache 2.2.11 that has a query string on it. The output is different if the query string is different, so I'm not ignoring the query string. My problem is that when I use the mem_cache module for this and look at the debug log, the effects seem different than when I use the disk_cache module. For mem_cache on every request it shows it getting cached again (and not getting served out of cache) or so that's how I'm reading it. For disk_cache it seems to cache it fine and serve out of cache on subsequent requests. Log with mem_cache (all requests): [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:47:23 2009] [info] mem_cache: Cached url: http://x.x.x.x/path?var=value Log with disk_cache (first request): [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(962): disk_cache: Stored headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(1051): disk_cache: Body for URL http://x.x.x.x/path?var=value cached. Log with disk_cache (subsequent request): [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(476): disk_cache: Recalled cached URL info header http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(749): disk_cache: Recalled headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(282): cache: running CACHE_OUT filter [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(296): cache: serving /path Response headers for content: DateFri, 20 Feb 2009 23:01:08 GMT Server Apache Cache-Control max-age=300, must-revalidate Expires Fri, 20 Feb 2009 23:06:08 GMT Connection close Transfer-Encoding chunked Content-Typetext/plain Content-LanguageEn Cache config (mem): CacheEnable mem /path CacheDefaultExpire 300 CacheIgnoreNoLastMod On CacheIgnoreQueryString Off CacheRoot /var/www/cache CacheDirLevels 1 CacheDirLength 4 CacheMaxFileSize 15 CacheMinFileSize 10 MCacheRemovalAlgorithm LRU MCacheMaxObjectCount 100 MCacheMaxObjectSize 15 MCacheMinObjectSize 10 MCacheSize 10240 Cache config (disk): Same as mem config except CacheEnable line is "CacheEnable disk /path" Any help is appreciated. -Tony ---
RE: mem_cache and disk_cache acting differently when determine when to cache content?
I stand corrected, The mem_cache module DOES serve out the cached version, but not until the 6th request of the URL. The disk_cache module serves out the cached version on the 2nd request (as it should). FYI, I'm using the worker mpm. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Anthony J. Biacco Sent: Friday, February 20, 2009 4:05 PM To: 'us...@httpd.apache.org' Cc: 'modules-dev@httpd.apache.org' Subject: mem_cache and disk_cache acting differently when determine when to cache content? Hi, I'm serving out dynamic content from apache 2.2.11 that has a query string on it. The output is different if the query string is different, so I'm not ignoring the query string. My problem is that when I use the mem_cache module for this and look at the debug log, the effects seem different than when I use the disk_cache module. For mem_cache on every request it shows it getting cached again (and not getting served out of cache) or so that's how I'm reading it. For disk_cache it seems to cache it fine and serve out of cache on subsequent requests. Log with mem_cache (all requests): [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:47:23 2009] [info] mem_cache: Cached url: http://x.x.x.x/path?var=value Log with disk_cache (first request): [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(962): disk_cache: Stored headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(1051): disk_cache: Body for URL http://x.x.x.x/path?var=value cached. Log with disk_cache (subsequent request): [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(476): disk_cache: Recalled cached URL info header http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(749): disk_cache: Recalled headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(282): cache: running CACHE_OUT filter [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(296): cache: serving /path Response headers for content: DateFri, 20 Feb 2009 23:01:08 GMT Server Apache Cache-Control max-age=300, must-revalidate Expires Fri, 20 Feb 2009 23:06:08 GMT Connection close Transfer-Encoding chunked Content-Typetext/plain Content-LanguageEn Cache config (mem): CacheEnable mem /path CacheDefaultExpire 300 CacheIgnoreNoLastMod On CacheIgnoreQueryString Off CacheRoot /var/www/cache CacheDirLevels 1 CacheDirLength 4 CacheMaxFileSize 15 CacheMinFileSize 10 MCacheRemovalAlgorithm LRU MCacheMaxObjectCount 100 MCacheMaxObjectSize 15 MCacheMinObjectSize 10 MCacheSize 10240 Cache config (disk): Same as mem config except CacheEnable line is "CacheEnable disk /path" Any help is appreciated. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com
mem_cache and disk_cache acting differently when determine when to cache content?
Hi, I'm serving out dynamic content from apache 2.2.11 that has a query string on it. The output is different if the query string is different, so I'm not ignoring the query string. My problem is that when I use the mem_cache module for this and look at the debug log, the effects seem different than when I use the disk_cache module. For mem_cache on every request it shows it getting cached again (and not getting served out of cache) or so that's how I'm reading it. For disk_cache it seems to cache it fine and serve out of cache on subsequent requests. Log with mem_cache (all requests): [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:47:23 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:47:23 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:47:23 2009] [info] mem_cache: Cached url: http://x.x.x.x/path?var=value Log with disk_cache (first request): [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(131): Adding CACHE_SAVE filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(138): Adding CACHE_REMOVE_URL filter for /path [Fri Feb 20 15:55:18 2009] [debug] mod_headers.c(740): headers: ap_headers_output_filter() [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(639): cache: Caching url: /path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_cache.c(645): cache: Removing CACHE_REMOVE_URL filter. [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(962): disk_cache: Stored headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:55:18 2009] [debug] mod_disk_cache.c(1051): disk_cache: Body for URL http://x.x.x.x/path?var=value cached. Log with disk_cache (subsequent request): [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(476): disk_cache: Recalled cached URL info header http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_disk_cache.c(749): disk_cache: Recalled headers for URL http://x.x.x.x/path?var=value [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(282): cache: running CACHE_OUT filter [Fri Feb 20 15:56:42 2009] [debug] mod_cache.c(296): cache: serving /path Response headers for content: DateFri, 20 Feb 2009 23:01:08 GMT Server Apache Cache-Control max-age=300, must-revalidate Expires Fri, 20 Feb 2009 23:06:08 GMT Connection close Transfer-Encoding chunked Content-Typetext/plain Content-LanguageEn Cache config (mem): CacheEnable mem /path CacheDefaultExpire 300 CacheIgnoreNoLastMod On CacheIgnoreQueryString Off CacheRoot /var/www/cache CacheDirLevels 1 CacheDirLength 4 CacheMaxFileSize 15 CacheMinFileSize 10 MCacheRemovalAlgorithm LRU MCacheMaxObjectCount 100 MCacheMaxObjectSize 15 MCacheMinObjectSize 10 MCacheSize 10240 Cache config (disk): Same as mem config except CacheEnable line is "CacheEnable disk /path" Any help is appreciated. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com
RE: cache POST requests
Also becomes more of a problem when my page's JS would have to know for subsequent requests how to get the cached version, and not simply do a PRG again, because it would not know if a cached version existed. Even if I sent a probing GET request pre-all reformatting requests, the JS would not have the hash. And now I'd be doing 3 requests for every 1 I was doing beforehand. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message----- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Tuesday, February 10, 2009 2:53 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests Ok, then the /URL/1 part becomes the problem for me. I can make the GET redirect request I send back to have the same key pairs in it to retrieve the same content and cache it, but again, then the redirect GET is longer than 2k (or however long the POST content was, just in the GET format), and I run into the IE truncate problem. I'd have to construct a URL that's: 1. <= 2k and 2. Has enough uniqueness that no 2 gets for different content are the same. 3. Has some way of tieing that get url into the same POST content. Something like, POST the content, have the app server MD5 hash the content (which if it's large, like 1MB put a strain on the app-server), put the content and hash somewhere (app server, db, memcached), respond with the redirect GET url including the hash in it, then when the redirect request comes back to the app server, recognize a hash is in it, and have it pull the content for that hash, and then apache subsequently would cache based on that URL with the hash in it. MIGHT be work the extra request and hashing time/storage. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Tuesday, February 10, 2009 2:24 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests Is this a little clearer? Notice how the Get request and post response are both tiny. This also opens up a whole slew of other possibilities now that you have a per-request URL to access. Current (as I understand it): -> POST (DOC file #1, 800KB) <- 200 OKAY (PDF file 600KB - non-cacheable, and substantial usability problems) Alternative (post, redirect, get): -> POST (DOC file #1, 800KB) <- REDIRECT FOR GET /URL/1 (< 1KB) (non-cacheable) -> GET /URL/1 (< 1KB) <- 200 OKAY (PDF file 600KB - easily cacheable, options for bookmarking, etc.) Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Tuesday, February 10, 2009 3:46 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests I did mean requests, yes. We run a content reformatting service using Tomcat, so in reality the responses are large also, because nearly the same content (formatted differently) is sent back. Another problem I found with large GETs is that IE will truncation them if they are 2k or larger. Now, if that is pre- rendering engine or in the actual engine, I don't know? If it's pre then maybe the limit wouldn't be affected a redirect? That gets a little off-topic. Thanx, -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Tuesday, February 10, 2009 12:20 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests You mean post REQUESTS, not RESPONSES, correct? GET requests shouldn't be very large, but it's not all that uncommon to have GET responses larger than 1GB (local LANS, etc.). Accept all the incomming data on a post (which could be 1+MB file attachments, etc.), generate a unique URL, and redirect the user there to fetch the result. Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Tuesday, February 10, 2009 1:52 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests I read that for the 1.0 spec, but thought for the 1.1 it was possible with the proper expiration headers. Although I do understand the keying problem. My problem is that my POSTs vary wildly in size from 5k to over a meg, and avg. out to about 45k. Being that GETs in apache by default are limited to 8k, I'll get a 414 error, so I'm not sure where I can turn to cache this. I suppose I can up the LimitRequestLine parameter to the max I need, but I'm not sure how kosher that is. Thanx, -Tony ---
RE: cache POST requests
Ok, then the /URL/1 part becomes the problem for me. I can make the GET redirect request I send back to have the same key pairs in it to retrieve the same content and cache it, but again, then the redirect GET is longer than 2k (or however long the POST content was, just in the GET format), and I run into the IE truncate problem. I'd have to construct a URL that's: 1. <= 2k and 2. Has enough uniqueness that no 2 gets for different content are the same. 3. Has some way of tieing that get url into the same POST content. Something like, POST the content, have the app server MD5 hash the content (which if it's large, like 1MB put a strain on the app-server), put the content and hash somewhere (app server, db, memcached), respond with the redirect GET url including the hash in it, then when the redirect request comes back to the app server, recognize a hash is in it, and have it pull the content for that hash, and then apache subsequently would cache based on that URL with the hash in it. MIGHT be work the extra request and hashing time/storage. -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Tuesday, February 10, 2009 2:24 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests Is this a little clearer? Notice how the Get request and post response are both tiny. This also opens up a whole slew of other possibilities now that you have a per-request URL to access. Current (as I understand it): -> POST (DOC file #1, 800KB) <- 200 OKAY (PDF file 600KB - non-cacheable, and substantial usability problems) Alternative (post, redirect, get): -> POST (DOC file #1, 800KB) <- REDIRECT FOR GET /URL/1 (< 1KB) (non-cacheable) -> GET /URL/1 (< 1KB) <- 200 OKAY (PDF file 600KB - easily cacheable, options for bookmarking, etc.) Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -----Original Message- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Tuesday, February 10, 2009 3:46 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests I did mean requests, yes. We run a content reformatting service using Tomcat, so in reality the responses are large also, because nearly the same content (formatted differently) is sent back. Another problem I found with large GETs is that IE will truncation them if they are 2k or larger. Now, if that is pre- rendering engine or in the actual engine, I don't know? If it's pre then maybe the limit wouldn't be affected a redirect? That gets a little off-topic. Thanx, -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Tuesday, February 10, 2009 12:20 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests You mean post REQUESTS, not RESPONSES, correct? GET requests shouldn't be very large, but it's not all that uncommon to have GET responses larger than 1GB (local LANS, etc.). Accept all the incomming data on a post (which could be 1+MB file attachments, etc.), generate a unique URL, and redirect the user there to fetch the result. Thanks, Rick Houser Auto-Owners Insurance Systems Support (517)703-2580 -Original Message- From: Anthony J. Biacco [mailto:abia...@formatdynamics.com] Sent: Tuesday, February 10, 2009 1:52 PM To: modules-dev@httpd.apache.org Subject: RE: cache POST requests I read that for the 1.0 spec, but thought for the 1.1 it was possible with the proper expiration headers. Although I do understand the keying problem. My problem is that my POSTs vary wildly in size from 5k to over a meg, and avg. out to about 45k. Being that GETs in apache by default are limited to 8k, I'll get a 414 error, so I'm not sure where I can turn to cache this. I suppose I can up the LimitRequestLine parameter to the max I need, but I'm not sure how kosher that is. Thanx, -Tony --- Manager, IT Operations Format Dynamics, Inc. 303-573-1800x27 abia...@formatdynamics.com http://www.formatdynamics.com -Original Message- From: Houser, Rick [mailto:houser.r...@aoins.com] Sent: Tuesday, February 10, 2009 11:37 AM To: modules-dev@httpd.apache.org; us...@httpd.apache.org Subject: RE: cache POST requests You really shouldn't be trying to cache responses to post requests. Completely from memory, but the HTTP spec says not to cache post responses. The URI is the base key to any caching implementations (with the addition of a select few vary headers, etc.), and your post data really doesn't factor in. The normal pattern to use in most of these situations is http://en.wikipedia.org/wiki/Post/Redirect/Get. Th
mod_rewrite bug/by design with mod_mem_cache cached files?
Hi,
I'm using apache 2.2.9 with mod_mem_cache to cache some javascript
files, which is working fine.
Today, I added a rewrite rule to forbid access (403) to everything in a
virtualhost based on the Opera browser, which obviously included the
javascript files.
Problem is, the cached files in memory weren't getting forbidden, yet
everything non-cached was.
If I turn off mod_mem_cache, everything gets forbidden as it should.
Is there either a bug in the rewrite or caching modules that is allowing
this rule to get circumvented? Is it by design? Am I messing up my
config?
Here's what I have in my virtualhost config..
RewriteCond %{HTTP_USER_AGENT} ^.*(opera).*$ [NC]
RewriteRule ^(.*)$ - [F,L]
CacheDefaultExpire 300
CacheEnable mem /js
CacheEnable mem /images
CacheIgnoreQueryString On
MCacheRemovalAlgorithm LRU
MCacheMaxObjectCount 100
MCacheMaxObjectSize 15
MCacheMinObjectSize 10
MCacheSize 10240
Here are sample back-to-back log entries (note, last parameter in log
line is %{Age}o"
129.7.109.142 - - [05/Sep/2008:08:13:25 -0600] "GET /js/2273/pd.js?ps=6
HTTP/1.1" 200 7545 "REFERER" "Opera/9.50 (Windows NT 5.1; U; en)" 1
129.7.109.142 - - [05/Sep/2008:08:13:27 -0600] "GET
/cp/psj?killCache=1220624007222&divId=2273&printSpecId=6 HTTP/1.1" 403
15 "REFERER" "Opera/9.50 (Windows NT 5.1; U; en)" -
-Tony
---
Manager, IT Operations
Format Dynamics, Inc.
303-573-1800x27
[EMAIL PROTECTED]
http://www.formatdynamics.com
---
Manager, IT Operations
Format Dynamics, Inc.
303-573-1800x27
[EMAIL PROTECTED]
http://www.formatdynamics.com
