Module caching
Hello all, I am working on a large modperl app, and one of the features of this is a plugin system that allows others to write and install modules. Everything is good as far as this goes, but the problem is updateing/deleting modules. It seems as though the code is cached until an apache restart (i.e code changes don't take effect, version numbers don't change). Is there a way to flush the INC hash of all the children programmatically, without a restart? I have looked at Apache::Reload and Apache::StatINC and tried to replicate the code, but it doesn't seem to be working. Thanks, Scott
Re: Module caching
On Fri, 2003-08-01 at 13:42, Scott wrote: I have looked at Apache::Reload and Apache::StatINC And what was wrong with them? You should know that there is no perfect way to reload a Perl module. It just isn't a feature of the language. Those two modules come as close as you can get without actually starting a new interpreter, but it is still possible for some code (especially code using closures) to interact badly with them. - Perrin
Re: Module caching
Scott [EMAIL PROTECTED] writes: Hello all, I am working on a large modperl app, and one of the features of this is a plugin system that allows others to write and install modules. Everything is good as far as this goes, but the problem is updateing/deleting modules. It seems as though the code is cached until an apache restart (i.e code changes don't take effect, version numbers don't change). Is there a way to flush the INC hash of all the children programmatically, without a restart? I have looked at Apache::Reload and Apache::StatINC and tried to replicate the code, but it doesn't seem to be working. The best thing I happened to meet here: - apache compiled with mod_perl as DSO (for instance debian linux version) - graceful restart which is invisible for the clients but reloads perl interpreter when mod_perl is DSO
Re: caching questions
On Tue, 22 Jul 2003 13:05:00 +0300, Stas Bekman wrote: Peter Haworth wrote: Cache::Mmap (which I wrote) isn't threadsafe, but should work OK in a forking MPM. If anyone wants to contribute code to make it threadsafe (preferably without impacting single-threaded performance too much), I'll be very grateful. I'd do it myself, but I've never used threads, so I'm bound to get it wrong. I haven't looked at the module's implementation, but if you have a C-level thread-safety issues, you may consider using libapr which provides a thread-safe mmap API. No, it's perl-level stuff at the point that matters. Currently, it uses fine grained file locking to prevent concurrent accesses from other processes, but as far as I understand it, this won't prevent concurrent accesses from other threads in the same process. -- Peter Haworth [EMAIL PROTECTED] Do you expect me to implement it? No, Mister Schwern, I expect you to DIE! -- Michael G Schwern in p5p
Re: caching questions
Sorry to step in here, but could I use any of the caching modules you mentionned in mod_perl2? thx tom Am Mon, 2003-07-21 um 21.45 schrieb Perrin Harkins: On Sun, 2003-07-20 at 15:47, Patrick Galbraith wrote: One thing that my code does is check to see if it's cache has been updated to the db, which I wish I could really have some sort of cache that's in one place in memory (as opposed to each child process). Well, you can't, unless you use threads. I know there's IPC, and also memcached. I've used IPC::Shareable before, but don't know if that's the route I'd wanna go. It's not. IPC::Shareable is very slow. If you want to share data, use Cache::FileCache, Cache::Mmap, MLDBM::Sync, or IPC::MM. Has anyone implemented a good caching system that sets up a global cache that you don't have to check constantly because you happen to have been served out by a child that doesn't have the latest cache from DB? One single memory object? The systems I listed above all use a shared cache that will have the same data regardless of which process accesses it. - Perrin
Re: caching questions
On Tue, 2003-07-22 at 02:13, Tom Schindl wrote: Sorry to step in here, but could I use any of the caching modules you mentionned in mod_perl2? I can't vouch for the thread safety of these modules, but all of them should work in prefork mode. - Perrin
Re: mod_perl caching form data?
This appears to be the classic global variable/uninitialized variable issue with your script. Mod_perl will load that script once and never reload it again unless you tell it too, even when different users access that script. If you have written a CGI script that doesn't lead itself to a ready rewrite to run under mod_perl, try running it under the PerlRun mode of mod_perl, it works great for this situation. Go here to read up on the issue: http://perl.apache.org/docs/1.0/guide/porting.html#Sometimes_it_Works__Sometimes_it_Doesn_t dale - Original Message - From: David Ressman [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, May 29, 2003 1:19 PM Subject: mod_perl caching form data? Hi all, I'm having a problem with mod_perl 1.2.7 that's baffling me completely, and I've been searching and reading for days, but I still can't figure it out. My apologies if this comes up frequently. I did try rather lengthy searches through the mailing list archives. Right now, I'm using mod_perl 1.2.7 compiled into an apache 1.3.27 server running on a Solaris 9 (semi-current patches, though I can't imagine that that's relevant) server. I've written some fairly straight-forward mod_perl scripts (using CGI.pm). They take form data from the user, process it, and stick it in a database (through DBI). So far, so good. Everything works pretty well... Except that something's caching previously entered form data and displaying it back to me as the default values in those same forms. As an example, this form has a text field that asks for IP addresses, and the text input will occasionally be filled out with the IP address of a system that you had entered a few minutes ago. Naturally, I suspected that my browser was the guilty party, even though I had specified '-1d' as the expiration time in the CGI header() command. It turns out that this is not the case. The forms will occasionally be pre-filled out with IP addresses that other people have given! I even went so far as to set up a network sniffer to verify that the server was indeed setting (in the HTML it sent to the client) the value parameter of the text fields to an IP address that another user had previously entered. Needless to say, my script is *not* setting the default or value parameters for these text fields. As an uneducated guess, I'd say that each httpd child-process is automatically filling out forms with data that it itself has previously received, but that's only a guess, and it still doesn't get me any closer to figuring out why it's happening. Can anyone offer any assistance or point me somewhere that I could find some documentation on what's happening? I'm completely baffled. Thanks! David
Re: mod_perl caching form data?
Perrin == Perrin Harkins [EMAIL PROTECTED] writes:
Perrin On Thu, 2003-05-29 at 17:26, [EMAIL PROTECTED] wrote:
A simple $cgi-delete('ipaddress') to delete the value when I create
the field has done the trick. Thanks very much to the both of you.
Perrin I'm glad to hear that worked, but it's still worrisome that you were
Perrin seeing data leak between different users. The form widgets are only
Perrin supposed to be sticky for values submitted on the current request. It
Perrin indicates either a bug in the way CGI.pm clears its variables under
Perrin mod_perl or a bug in your script that could surface again later.
OK, throwing my hat into the ring here...
maybe *I* wasn't hallucinating then. I just had a recent
fix to a longstanding bug in the picture section of my website...
I had been using CGI.pm (through Template::Plugin::CGI), and was
mystified because *occasionally* the wrong picture would show, but a
simple reload fixed it.
I fixed the bug by avoiding CGI.pm, and using Apache::Template's
param variable directly instead.
So maybe there is a CGI.pm bug with regarding to clearing out the
values in a mod_perl environment. I wonder how simple of a test we
can concoct to determine that?
Lincoln, are you listening?
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
[EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: mod_perl caching form data?
On Fri, 2003-05-30 at 10:42, Randal L. Schwartz wrote: I had been using CGI.pm (through Template::Plugin::CGI), and was mystified because *occasionally* the wrong picture would show, but a simple reload fixed it. I fixed the bug by avoiding CGI.pm, and using Apache::Template's param variable directly instead. So maybe there is a CGI.pm bug with regarding to clearing out the values in a mod_perl environment. I wonder how simple of a test we can concoct to determine that? Let's not jump to conclusions. We don't have a test case yet, and no one has seen David's code so he might have a simple global variables problem. Many people are using CGI.pm successfully with mod_perl, so if there's a problem it only happens in specific circumstances. If someone has a test case, by all means, post it. - Perrin
Re: mod_perl caching form data?
Thus spake Perrin Harkins ([EMAIL PROTECTED]): Let's not jump to conclusions. We don't have a test case yet, and no one has seen David's code so he might have a simple global variables problem. I suspect that the more likely explanation of this particular problem is that I goofed up. Anyway, the script is a thousand or so lines long, so it doesn't make a whole lot of sense to post it here. I'll go through it all with a fine-toothed comb and try to see if I can make the problem go away without having to call delete() to wipe out the parameters. If I'm able to do it, I'll certainly send something back to the list.
mod_perl caching form data?
[Sorry if an identical message comes through later. I don't think the list manager liked my address the first time I sent it through.] Hi all, I'm having a problem with mod_perl 1.2.7 that's baffling me completely, and I've been searching and reading for days, but I still can't figure it out. My apologies if this comes up frequently. I did try rather lengthy searches through the mailing list archives. Right now, I'm using mod_perl 1.2.7 compiled into an apache 1.3.27 server running on a Solaris 9 (semi-current patches, though I can't imagine that that's relevant) server. I've written some fairly straight-forward mod_perl scripts (using CGI.pm). They take form data from the user, process it, and stick it in a database (through DBI). So far, so good. Everything works pretty well... Except that something's caching previously entered form data and displaying it back to me as the default values in those same forms. As an example, this form has a text field that asks for IP addresses, and the text input will occasionally be filled out with the IP address of a system that you had entered a few minutes ago. Naturally, I suspected that my browser was the guilty party, even though I had specified '-1d' as the expiration time in the CGI header() command. It turns out that this is not the case. The forms will occasionally be pre-filled out with IP addresses that other people have given! I even went so far as to set up a network sniffer to verify that the server was indeed setting (in the HTML it sent to the client) the value parameter of the text fields to an IP address that another user had previously entered. Needless to say, my script is *not* setting the default or value parameters for these text fields. As an uneducated guess, I'd say that each httpd child-process is automatically filling out forms with data that it itself has previously received, but that's only a guess, and it still doesn't get me any closer to figuring out why it's happening. Can anyone offer any assistance or point me somewhere that I could find some documentation on what's happening? I'm completely baffled. Thanks! David
Re: mod_perl caching form data?
Quoting David Ressman [EMAIL PROTECTED]: something's caching previously entered form data and displaying it back to me as the default values in those same forms. As an example, this form has a text field that asks for IP addresses, and the text input will occasionally be filled out with the IP address of a system that you had entered a few minutes ago. If you do a view source in the browser, and check the form fields, do they have the VALUE=... paramters set? ie do you see something like this: input type=text name=ipaddress value=192.168.1.1 If that value=192.168.1.1 is set in any of your form fields, then your script is the cuprit. If your form fields don't have the value=192.168.1.1 set, or it is set to value=, then your browser is the culprit. Most modern browsers will offer to remember values you have entered into a form. On subsequent visits to the same URL, the values will be prefilled by the browser for you. mod_perl itself will never prefill form values for you. It is completely up to your script (or the modules it uses) to munge the data coming through. It is possible that you are creating your form fields with CGI.pm, which will use the currently POSTed parameters to prefill the form, or if you are using a module like HTML::FillInForm, which will also prefill form fields. But you would have to explicitly use these features in your script. HTH Cees,
Re: mod_perl caching form data?
Thus spake Cees Hek ([EMAIL PROTECTED]): input type=text name=ipaddress value=192.168.1.1 If that value=192.168.1.1 is set in any of your form fields, then your script is the cuprit. If your form fields don't have the value=192.168.1.1 set, or it is set to value=, then your browser is the culprit. That's just the problem. The value= parameter *is* filled out (and it is being done at the server end as evidenced by the network sniffer.) Unfortunately, my script is not doing it. Here's what I have in the script: print IP Address: , $cgi-textfield(-name='ipaddress', -size=20, -maxlength=20), $cgi-p; But yet the form is filled out when I go to it with a web browser. Most modern browsers will offer to remember values you have entered into a form. On subsequent visits to the same URL, the values will be prefilled by the browser for you. I was certain that that's what it was, but it just can't be. I've even used browsers on computers that have never been to the site before and the data is still pre-filled out. mod_perl itself will never prefill form values for you. It is completely up to your script (or the modules it uses) to munge the data coming through. That's good to know. Thanks very much. It is possible that you are creating your form fields with CGI.pm, which will use the currently POSTed parameters to prefill the form, or This sounds like the most likely culprit, even though I haven't explicitly turned anything on. It's possible that I've done it inadvertently. I'll check it out. Thanks again for your help. David
Re: mod_perl caching form data?
Quoting David Ressman [EMAIL PROTECTED]: It is possible that you are creating your form fields with CGI.pm, which will use the currently POSTed parameters to prefill the form, or This sounds like the most likely culprit, even though I haven't explicitly turned anything on. It's possible that I've done it inadvertently. I'll check it out. Read the CGI.pm docs and you will find the cause of your problem: http://search.cpan.org/author/JHI/perl-5.8.0/lib/CGI.pm#CREATING_FILL_OUT_FORMS_ Cheers, Cees
Re: mod_perl caching form data?
On Thu, 2003-05-29 at 16:40, David Ressman wrote: That's just the problem. The value= parameter *is* filled out (and it is being done at the server end as evidenced by the network sniffer.) Unfortunately, my script is not doing it. Here's what I have in the script: print IP Address: , $cgi-textfield(-name='ipaddress', -size=20, -maxlength=20), $cgi-p; CGI.pm uses sticky widgets by default. These are supposed to be cleared between requests though, by a cleanup handler that CGI.pm uses. Are you using mod_perl 2, by any chance? I think I see a bug in CGI.pm's handling of this for mod_perl 2. It's looking for Apache-request, which doesn't work unless you have Apache::compat on. You might also run into problems if you are copying the CGI object into a global or accidentally creating a closure with it. If you can post a very small test case that exhibits the problem, we can probably tell you if you have either of those issues. - Perrin
Re: mod_perl caching form data?
Thus spake Perrin Harkins ([EMAIL PROTECTED]):
CGI.pm uses sticky widgets by default. These are supposed to be cleared
between requests though, by a cleanup handler that CGI.pm uses. Are you
using mod_perl 2, by any chance? I think I see a bug in CGI.pm's
Nope. mod_perl 1.27
But... you and Cees are beautiful people. I'd read almost completely
through the CGI.pm documentation, but I didn't remember that it was
significant that each http child only compiles the code once, so these
cgi parameters might stay in memory as new clients hit the children.
A simple $cgi-delete('ipaddress') to delete the value when I create
the field has done the trick. Thanks very much to the both of you.
Re: mod_perl caching form data?
On Thu, 2003-05-29 at 17:26, [EMAIL PROTECTED] wrote:
A simple $cgi-delete('ipaddress') to delete the value when I create
the field has done the trick. Thanks very much to the both of you.
I'm glad to hear that worked, but it's still worrisome that you were
seeing data leak between different users. The form widgets are only
supposed to be sticky for values submitted on the current request. It
indicates either a bug in the way CGI.pm clears its variables under
mod_perl or a bug in your script that could surface again later.
- Perrin
mod_perl caching form data?
Hi all, I'm having a problem with mod_perl 1.2.7 that's baffling me completely, and I've been searching and reading for days, but I still can't figure it out. My apologies if this comes up frequently. I did try rather lengthy searches through the mailing list archives. Right now, I'm using mod_perl 1.2.7 compiled into an apache 1.3.27 server running on a Solaris 9 (semi-current patches, though I can't imagine that that's relevant) server. I've written some fairly straight-forward mod_perl scripts (using CGI.pm). They take form data from the user, process it, and stick it in a database (through DBI). So far, so good. Everything works pretty well... Except that something's caching previously entered form data and displaying it back to me as the default values in those same forms. As an example, this form has a text field that asks for IP addresses, and the text input will occasionally be filled out with the IP address of a system that you had entered a few minutes ago. Naturally, I suspected that my browser was the guilty party, even though I had specified '-1d' as the expiration time in the CGI header() command. It turns out that this is not the case. The forms will occasionally be pre-filled out with IP addresses that other people have given! I even went so far as to set up a network sniffer to verify that the server was indeed setting (in the HTML it sent to the client) the value parameter of the text fields to an IP address that another user had previously entered. Needless to say, my script is *not* setting the default or value parameters for these text fields. As an uneducated guess, I'd say that each httpd child-process is automatically filling out forms with data that it itself has previously received, but that's only a guess, and it still doesn't get me any closer to figuring out why it's happening. Can anyone offer any assistance or point me somewhere that I could find some documentation on what's happening? I'm completely baffled. Thanks! David
RE: mod_perl caching form data?
David Ressman wrote:
something's caching previously entered form data and
displaying it back to me as the default values in those same forms
This is most likely a variable scoping problem as described here:
http://perl.apache.org/docs/1.0/guide/frequent.html
As a rule of thumb, variables which are 'local' to a subroutine
should be declared with 'my' and variables which should be
visible to all routines in a file should be declared with 'our'.
eg:
use CGI;
our $cgi = new CGI;
...
sub do_something {
my($arg1, $arg2) = @_;
...
}
If your script has global variables which are declared with 'my',
they will cause problems.
Regards
Grant
Re: mod_perl caching form data?
Hi!
On Thu, May 29, 2003 at 12:19:49PM -0500, David Ressman wrote:
So far, so good. Everything works pretty well... Except that
something's caching previously entered form data and displaying it back
to me as the default values in those same forms. As an example, this
form has a text field that asks for IP addresses, and the text input
will occasionally be filled out with the IP address of a system that
you had entered a few minutes ago.
Sounds to me like you are storing your values in global variables which
won't get reset between requests.
Do you declare you vars with 'my' ?
Are you running with
use strict;
use warnings;
? If not, turn them on, as thos pragmas will help you catch this kind of
error.
See here:
http://perl.apache.org/docs/1.0/guide/porting.html#Global_Variables_Persistence
--
#!/usr/bin/perl http://domm.zsi.at
for(ref bless{},just'another'perl'hacker){s-:+-$-gprint$_.$/}
Caching with mod_proxy reverse proxy on Apache 1.3.x
(I sent this initially to the apache mailing list, but nobody has replied... so I am hoping that someone who is using mod_perl has seen this problem and knows the solution...) Hi, I have set up my server to use mod_proxy as a reverse proxy front end for mod_perl. There are two builds of apache, both on the same machine. Setup: RedHat 7.3, fully updated. Single machine, with two servers: Front end: Apache/1.3.27 (Unix) mod_ssl/2.8.14 OpenSSL/0.9.6b Back end: Apache/1.3.27 (Unix) mod_gzip/1.3.19.1a mod_perl/1.27 The purpose is to relieve load on the mod_perl server. The front end serves all images and static pages, the mod_perl apache serves all dynamic html pages. This is working just fine at the moment, but I have never been able to get mod_proxy to cache any pages served from the mod_perl server. I have set up all the variables specified in the mod_proxy documentation (i.e. CacheRoot), but html pages do not get cached. I have tried turning off mod_gzip, and made sure that the mod_perl server is generating correct ContentLength and Expires headers. But nothing ever gets cached. The reason I would like this is that some of the pages on my sites are dynamic, but don't change much. They would benefit from being cached at least for a while, so that the mod_perl server doesn't have to regenerate them each and every time. This takes load off the SQL server too, which also gets hit every time. The proxy setup uses mod_rewrite to do the proxying: RewriteEngine On RewriteOptions 'inherit' RewriteRule \.(gif|jpg|png|ico|htm|cgi)$ - [last] RewriteRule ^/(.*)$ http://localhost:8080/$1 [proxy] ProxyReceiveBufferSize 131072 # Reverse proxy cache does not work! CacheRoot /www/vhosts/www.neilgunton.com/cache CacheSize 1024 The mod_perl server then has the VirtualHost set up to catch the forwarded requests. As I said, everything works perfectly except that mod_proxy never caches anything. The one clue I have is this: I have noticed during experimentation that if I use a separate machine for the front end proxy, and forward the mod_perl requests to the other machine, then it does seem to cache. So does mod_proxy never cache anything that is coming from the same machine (i.e. localhost)? If so then this looks like an oversight, since it can be very useful to cache even local content if it is being dynamically generated. I know that there is a new caching module for Apache 2.0, but I am not ready to upgrade yet and am otherwise quite satisfied with 1.3.x currently. I am just wondering if there is some magic switch I can throw for mod_proxy to cache mod_perl content coming from localhost. I have tried changing the redirect to use the fully specified domain instead of localhost, but that doesn't seem to work at all (I get a 404 error). Any clues much appreciated. Thanks in advance, -Neil
cookie authenticated caching proxy
Hi, I am still pretty new to mod_perl and have only really dabbled with a few small applications that I created. I am now looking to build something quite specific and was wondering if anyones know of anything similar to what I want to do so or whether they have any pointers as to mod_perl's suitability for what I want to do. The scenario: I have a large web-site that is largely database driven. Many of the pages are accessed frequently yet are only modified on average once a day. The site is written in a variety of technologies including PHP and Perl. I wanted to develop a caching proxy that will return a cached page instead of passing control to one of the PHP scripts or Perl scripts that normally generate the pages. The web site is mainly accessed through HTTP GET requests rather than POST requests and the resultant page will be the same each time if given the same GET string. Can I use mod_perl's proxy capabilities to cache the generated contents of the the HTML page made by the PHP script? One extra requirement is that some of the pages need authentication which is cookie based. I would need to check the cookie against a database to see if the caller is authenticated to access the particular page. Has anyone done/seen anything similar implemented in mod_perl? Thanks for your help, Abdul-Wahid signature.asc Description: This is a digitally signed message part
Re: cookie authenticated caching proxy
Abdul-wahid Paterson wrote: I wanted to develop a caching proxy that will return a cached page instead of passing control to one of the PHP scripts or Perl scripts that normally generate the pages. This is called a reverse proxy and is very common in mod_perl setups. It is typically done with the mod_proxy module and mod_rewrite. See this for more details: http://perl.apache.org/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode One extra requirement is that some of the pages need authentication which is cookie based. I would need to check the cookie against a database to see if the caller is authenticated to access the particular page. Has anyone done/seen anything similar implemented in mod_perl? That's very easy to do in mod_perl, but you generally do not want to run mod_perl on the proxy server. The idea is to keep the proxy server really small and light. There are various auth modules for apache written in C which you might be able to use. If none of those suit your needs and you don't want to write a C module, you certainly can put mod_perl on the proxy and write what you want in Perl. The only drawback to this is the additional memory that will be needed for that server. - Perrin
Re: cookie authenticated caching proxy
One extra requirement is that some of the pages need authentication which is cookie based. I would need to check the cookie against a database to see if the caller is authenticated to access the particular page. Has anyone done/seen anything similar implemented in mod_perl? That's very easy to do in mod_perl, but you generally do not want to run mod_perl on the proxy server. The idea is to keep the proxy server really small and light. There are various auth modules for apache written in C which you might be able to use. If none of those suit your needs and you don't want to write a C module, you certainly can put mod_perl on the proxy and write what you want in Perl. The only drawback to this is the additional memory that will be needed for that server. In the docs you cited, it says: ProxyPass happens before the authentication phase, so you do not have to worry about authenticating twice. How does this work? I need to check the clients cookie against a database to make sure they have sufficient authorisation to access the cached page. The authentication mechanisms are going to be different depending on whether the page has already been cached. That is, if the page is coming from the dynamic webserver then that webserver is going to create the content and do the authentication/authorisation however for subsequent visitors I need the caching reverse proxy to do the authentication. Could you expand on the steps in the apache process cycle involved here? I can't quite see how this fits together. I would have thought that I would have to write my own authentication module as our authentication system is somewhat custom. I would probably prototype it in mod_perl then if needed re-write it in C. Thanks, Abdul-Wahid signature.asc Description: This is a digitally signed message part
Re: cookie authenticated caching proxy
Abdul-wahid Paterson wrote: In the docs you cited, it says: ProxyPass happens before the authentication phase, so you do not have to worry about authenticating twice. Hmmm, I thought you had an opportunity to do access control first. Look at this, from the mod_proxy docs: http://httpd.apache.org/docs/mod/mod_proxy.html#access That shows use of mod_access. With mod_access, you can deny all requests that don't have a particular environment variable set. You should be able to code a module which sets that environment variable based on your auth system. However, I could be wrong about this since I've never looked at the mod_proxy source. If mod_proxy won't work, it is possible to do the whole thing in Perl. Take a look at the modules listed here: http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?search=apache+proxynew=Searchfiletype=+distribution+name+or+descriptionsite=ftp.funet.fijoin=andstem=noarrange=filecase=clikedownload=autoage= - Perrin
Re: caching dynamic content in the reverse proxy
Good afternoon, On 6/9/02 at 12:12 AM, pascal barbedor [EMAIL PROTECTED] wrote: but any content sent by a modperl handler is not cached although I set headers Last-Modified and Expires to compatibles dates (with Apache::Util::ht_time) and Cache-content to public from the modperl handler with $r-header_out.. Has any one succeeeded is caching dynamic content in this configuration When I have caching problems, I run my pages through cachability to find out where they are failing. Check it out at: http://www.web-caching.com/cgi-web-caching/cacheability.py It reports on page headers as well as gives useful info like the following: This object doesn't have any explicit freshness information set, so a cache may use Last-Modified to determine how fresh it is with an adaptive TTL (at this time, it could be, depending on the adaptive percent used, considered fresh for: 2 weeks 20 hr (20%), 5 weeks 2 days (50%), 10 weeks 4 days (100%)). It can be validated with Last-Modified. The site is a great resource for a variety of caching issues. Charlie -- Charlie Garrison[EMAIL PROTECTED] PO Box 141, Windsor, NSW 2756, Australia
caching dynamic content in the reverse proxy
Hi following a recipe in modperl doc I have setup a front end plain apache "reverse proxy" + a back end modperl enabled. the front end has proxy cache enabled. any static content that is served by a default-handler from the backend server is well cached by the front one. but any content sent by a modperl handler is not cached although I set headers Last-Modified and Expires tocompatibles dates (with Apache::Util::ht_time) and Cache-content to public from the modperl handler with $r-header_out.. Also I wondered is the formatting of dates by ht_time (mer., 12 sept. 2002 12:20:01 GMT) by default is the "RFC one" ie the abbreviation point after day and month needed or not ? in any case i have tried both form without success for caching. Has any one succeeeded is caching dynamic content in this configuration thanks pascal winxp perl561/modperl126/apache1326
Caching in TT2 or Some Mod Perl oddity
I'm in a baffled state, ladies and gentleman.
I'm using TT2 to ultimately display some pages, however I'm having some
rather odd behaviour creeping in.
In my template constructor I have CACHE_SIZE set to 0.
I have a page which - even after a server restart - reverts to an older
version of the page.
Whilst I was still debugging, I accidently had Data::Dumper doing a print
STDOUT and a lovely $VAR1=[] in the corner of my screen. This has long
since been removed from my code, however I keep going through hits of the
same page - as I said, even after a restart - where either I have the
'correct' funky page, or on the next hit reverting to the old over
funked paged.
From my understanding of TT2, it's not the actual page which is cached but
some sort of compiled version of the nested code (?). Thus I'm having
trouble seeing how it might be a TT2 problem, especially with
{CACHE_SIZE=0} whenever Template is instantiated. Is it TT2 or is it
some Mod_Perlus Oddity?
Anyone?
Cheers,
Fiq
'Don't you love her madly?
Don't you need her badly?
Don't you love her ways?
Tell me what you say?
Don't you love her madly?
want to be her daddy?
Don't you love her face?
Don't you love her as she's walking out the door?'
- Jim Morrison, The DoorS.
Re: Caching in TT2 or Some Mod Perl oddity
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, Jun 26, 2002 at 11:17:49PM +, Rafiq Ismail (ADMIN) wrote: I have a page which - even after a server restart - reverts to an older version of the page. Whilst I was still debugging, I accidently had Data::Dumper doing a print STDOUT and a lovely $VAR1=[] in the corner of my screen. This has long since been removed from my code, however I keep going through hits of the same page - as I said, even after a restart - where either I have the 'correct' funky page, or on the next hit reverting to the old over funked paged. Sounds like browser caching, or a rather borked transparent proxy. Or both. - -- Stephen Clouse [EMAIL PROTECTED] Senior Programmer, IQ Coordinator Project Lead The IQ Group, Inc. http://www.theiqgroup.com/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE9GmXLA4aoazQ9p2cRAkFSAJ9tctOR4sNzmtWHhzyVaebftWMcxgCg1Uoh hlAtmfjtL+weqERfSqbCu5Q= =F7ij -END PGP SIGNATURE-
Anti caching and auto refresh problem
I'm setting the following headers with mod_perl and there seems to be a bug in the way netscape 4.7 handles them. The page is being auto refreshed by Netscape if the document in cache is compared to document on network preference is set to every time. I've tried IE and I don't seem to have the problem, but haven't had a chance to test on any other platforms. I've tried removing the Expires header, but that didn't make a difference. I've searched the archives and bug databases but nothing has turned up. Here is an example header: HTTP/1.1 200 OK Date: Wed, 12 Jun 2002 15:28:59 GMT Server: Apache/1.3.22 (Unix) mod_perl/1.26 Pragma: no-cache Cache-Control: no-cache Connection: close Content-Type: image/gif Expires: Wed, 12 Jun 2002 15:28:59 GMT I know it might be a bit off topic but I was wondering if anyone else has run into this problem? Thanks, Steve
Re: Anti caching and auto refresh problem
Steve Walker wrote: I'm setting the following headers with mod_perl and there seems to be a bug in the way netscape 4.7 handles them. The page is being auto refreshed by Netscape if the document in cache is compared to document on network preference is set to every time. I've tried IE and I don't seem to have the problem, but haven't had a chance to test on any other platforms. I've tried removing the Expires header, but that didn't make a difference. I've searched the archives and bug databases but nothing has turned up. Here is an example header: HTTP/1.1 200 OK Date: Wed, 12 Jun 2002 15:28:59 GMT Server: Apache/1.3.22 (Unix) mod_perl/1.26 Pragma: no-cache Cache-Control: no-cache Connection: close Content-Type: image/gif Expires: Wed, 12 Jun 2002 15:28:59 GMT I know it might be a bit off topic but I was wondering if anyone else has run into this problem? I'm not really sure what you're asking - are you saying that you _want_ NS to use its cache? if so, removing the Pragma and Cache-Control headers with $r-no_cache(0) should do that (and remove the Expires header as well, IIRC). If you _don't want_ netscape to use its cache, it looks like you're doing the right thing. There are some methods in Apache::File (like $r-set_last_modified) that you may want to look at as well in order to help clients make a more intelligent decision. HTH --Geoff
Re: Document Caching
On Wed, 6 Mar 2002, Rasoul Hajikhani wrote: Robert Landrum wrote: #startup.pl open(FILE,/path/to/tmpl); $MY::TEMPLATE .= while(FILE); close(FILE); Thanks... But I use Template Toolkit to generate a dynamic file. How would the above code work in that situation?! Not how I'd do it, although I'm pretty sure that TT2 has it's own caching mechanism, however in answer to this 'particular question' could you not overload TT's service object to use the preloaded instantiation similar, or more specialised than the one above. (SEE ALSO man Template::Service ..probably) :) fiq btw Apache::Template does let you control caching of templates in memory.
Re: Document Caching
Cahill, Earl wrote: I would not be opposed to calling a different, more standard function to check the cache (set up in a more standard way), and then fetch accordingly. Look at how the Memoize module does it. You may be able to do something similar that would allow caching to be added easilly to any pure virtual function. - Perrin
Document Caching
Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r
Re: Document Caching
At 12:28 PM -0800 3/6/02, Rasoul Hajikhani wrote: Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r #startup.pl open(FILE,/path/to/tmpl); $MY::TEMPLATE .= while(FILE); close(FILE); Provided that you never change $MY::TEMPLATE, this should work fine. -- When I used a Mac, they laughed because I had no command prompt. When I used Linux, they laughed because I had no GUI.
Re: Document Caching
Robert Landrum wrote: At 12:28 PM -0800 3/6/02, Rasoul Hajikhani wrote: Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r #startup.pl open(FILE,/path/to/tmpl); $MY::TEMPLATE .= while(FILE); close(FILE); Provided that you never change $MY::TEMPLATE, this should work fine. -- When I used a Mac, they laughed because I had no command prompt. When I used Linux, they laughed because I had no GUI. Thanks... But I use Template Toolkit to generate a dynamic file. How would the above code work in that situation?! -r
RE: Document Caching
Rasoul Hajikhani wrote: Robert Landrum wrote: At 12:28 PM -0800 3/6/02, Rasoul Hajikhani wrote: Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r #startup.pl open(FILE,/path/to/tmpl); $MY::TEMPLATE .= while(FILE); close(FILE); Provided that you never change $MY::TEMPLATE, this should work fine. -- When I used a Mac, they laughed because I had no command prompt. When I used Linux, they laughed because I had no GUI. Thanks... But I use Template Toolkit to generate a dynamic file. How would the above code work in that situation?! -r The Template Toolkit package includes it's own caching mechanism. Check out the pod for Template::Provider on your system. I'm not sure how it works under mod_perl, but it should be a good place for you to start. Tim
RE: Document Caching
Timothy Henigan wrote: At Wednesday, March 06, 2002 4:23 PM Rasoul Hajikhani wrote: Robert Landrum wrote: At 12:28 PM -0800 3/6/02, Rasoul Hajikhani wrote: Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r #startup.pl open(FILE,/path/to/tmpl); $MY::TEMPLATE .= while(FILE); close(FILE); Provided that you never change $MY::TEMPLATE, this should work fine. -- When I used a Mac, they laughed because I had no command prompt. When I used Linux, they laughed because I had no GUI. Thanks... But I use Template Toolkit to generate a dynamic file. How would the above code work in that situation?! -r The Template Toolkit package includes it's own caching mechanism. Check out the pod for Template::Provider on your system. I'm not sure how it works under mod_perl, but it should be a good place for you to start. Tim Slashdot caches templates under mod_perl in their system. For an example, download the latest version of slashcode (2.2.5) and take a look at the /slash/Slash/Display/Provider/Provider.pm module. http://sf.net/project/showfiles.php?group_id=4421release_id=73958 Tim
RE: Document Caching
I am finishing up a sort of alpha version of Data::Fallback (my own name) which should work very well for cache'ing just about anything locally on a box. We are planning on using it to cache dynamically generated html templates and images. You would ask a local perl daemon (using Net::Server) for the info and it would look first in the cache. If it isn't in the cache, it falls back according to where you told it to look (for now conffile or DBI, but later Storable, dbm, HTTP hit, whatever), and caches how you tell it to, based on ttl if you like. I am doing some testing now to see what sort of numbers we can get. Looking like 100-200 queries a second, but we'll see if that holds up in production, under high loads. I hope to write some docs on it over the weekend and get at least some alpha version CPAN'd before too long here. Earl -Original Message- From: Rasoul Hajikhani [mailto:[EMAIL PROTECTED]] Sent: Wednesday, March 06, 2002 1:28 PM To: [EMAIL PROTECTED] Subject: Document Caching Hello People, Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance -r
Re: Document Caching
Cahill, Earl wrote: I am finishing up a sort of alpha version of Data::Fallback (my own name) which should work very well for cache'ing just about anything locally on a box. We are planning on using it to cache dynamically generated html templates and images. You would ask a local perl daemon (using Net::Server) for the info and it would look first in the cache. If it isn't in the cache, it falls back according to where you told it to look (for now conffile or DBI, but later Storable, dbm, HTTP hit, whatever), and caches how you tell it to, based on ttl if you like. Hmmm... isn't that sort of backwards? It sounds like you're considering the problem as building a cache that can be taught how to fetch data, but to me it seems more natural to build components for fetching data and teach them how to cache. The semantic for describing how something can be cached are much simpler than those describing how something can be fetched. I would think it makes more sense to do something along the lines of the Memoize module, i.e. make it easy to add caching to your existing data fetching modules (hopefully using a standard interface like Cache::Cache). - Perrin
RE: Document Caching
Hmmm... isn't that sort of backwards? It sounds like you're considering the problem as building a cache that can be taught how to fetch data, but to me it seems more natural to build components for fetching data and teach them how to cache. The semantic for describing how something can be cached are much simpler than those describing how something can be fetched. I would think it makes more sense to do something along the lines of the Memoize module, i.e. make it easy to add caching to your existing data fetching modules (hopefully using a standard interface like Cache::Cache). Yeah, I buy that. Mostly I have been writing the fetching routines, and in sort of ad hoc fashion I have started to add on the caching stuff. I am just using a hash structure built on the modle File::CacheDir that I wrote. For me it is a two part problem that is pretty easily divisible. I have a function that checks the cache and if it returns false, then I fetch it according to the fallback. I would not be opposed to calling a different, more standard function to check the cache (set up in a more standard way), and then fetch accordingly. Earl
Re: Document Caching
Need your advise on how to cache a template under mod_perl... Any ideas? Thanks in advance Thanks... But I use Template Toolkit to generate a dynamic file. How would the above code work in that situation?! You should probably post this to the Template Toolkit mailing list for more info but TT has built-in support for caching templates to disk. TT mailing list: http://www.template-toolkit.org/info.html#lists TT caching: http://www.template-toolkit.org/docs/plain/Manual/Config.html#Caching_and_Compiling_Options --Ade. _ Do You Yahoo!? Get your free yahoo.com address at http://mail.yahoo.com
Caching LDAP connections
Hello Is there an Apache::Net::LDAP type module that would cache LDAP connections? Something that behaves in the same manner as Apache::DBI I went searching on cpan and sourceforge but didn't find what I was looking for. Thanks in advance. /--\ | Remi Godin | The greatest risk| | Metrex.net |is not taking one. | | (204) 943-0156 | \--/ Remi L Godin (E-mail).vcf Description: Binary data
Re: Caching LDAP connections
On Fri, 8 Feb 2002, Remi Godin wrote: Is there an Apache::Net::LDAP type module that would cache LDAP connections? Something that behaves in the same manner as Apache::DBI how about ResourcePool? it includes a ResourcePool::Resource::Net::LDAP class.
caching
How can kill If-Modified-Since and E-Tag incoming headers ? Do I do it in my custom PerlInitHandler module ? I have a development server and I don't wish anything to be cached (304 messages, not modified). Right now my Cookies are not set because of this. 10x
Re: caching
Miroslav Madzarevic wrote: How can kill If-Modified-Since and E-Tag incoming headers ? Do I do it in my custom PerlInitHandler module ? I have a development server and I don't wish anything to be cached (304 messages, not modified). Right now my Cookies are not set because of this. $r-no_cache(1); should help, anytime before calling $r-send_http_header() or prior to the content-generation phase (for files handled by http_core) --Geoff
Re: Comparison of different caching schemes
Ok, I'm a bit slow... At 03:05 PM 12/12/01 +1100, Rob Mueller (fastmail) wrote: >>>> Just thought people might be interested... Seems like they were! Thanks again. I didn't see anyone comment about this, but I was a bit surprised by MySQLs good performance. I suppose caching is key. I wonder if things would change with 50 or 100 thousand rows in the table. I always assumed something like Cache::FileCache would have less overhead than a RDMS. It's impressive. >>>> Now to the results, here they are. Package C0 - In process hash Sets per sec = 147116 Gets per sec = 81597 Mixes per sec = 124120 Package C1 - Storable freeze/thaw Sets per sec = 2665 Gets per sec = 6653 Mixes per sec = 3880 Package C2 - Cache::Mmap Sets per sec = 809 Gets per sec = 3235 Mixes per sec = 1261 Package C3 - Cache::FileCache Sets per sec = 393 Gets per sec = 831 Mixes per sec = 401 Package C4 - DBI with freeze/thaw Sets per sec = 651 Gets per sec = 1648 Mixes per sec = 816 Package C5 - DBI (use updates with dup) with freeze/thaw Sets per sec = 657 Gets per sec = 1994 Mixes per sec = 944 Package C6 - MLDBM::Sync::SDBM_File Sets per sec = 334 Gets per sec = 1279 Mixes per sec = 524 Package C7 - Cache::SharedMemoryCache Sets per sec = 42 Gets per sec = 29 Mixes per sec = 32 Bill Moseley mailto:[EMAIL PROTECTED]
Re: Comparison of different caching schemes
Rob Mueller (fastmail) wrote: And ++ on Paul's comments about Devel::DProf and other profilers. Ditto again. I've been using Apache::DProf recently and it's been great at tracking down exactly where time is spent in my program. One place that Rob and I still haven't found a good solution for profiling is trying to work out whether we should be focussing on optimising our mod_perl code, or our IMAP config, or our MySQL DB, or our SMTP setup, or our daemons' code, or... Can anyone suggest a way (under Linux 2.4, if it's OS dependent) to get a log of CPU (and also IO preferably) usage by process name over some period of time? Sorry to move slightly off-topic here, but in optimising our mod_perl app we have to work out which bit of the whole system to prioritise.
Re: Comparison of different caching schemes
One place that Rob and I still haven't found a good solution for profiling is trying to work out whether we should be focussing on optimising our mod_perl code, or our IMAP config, or our MySQL DB, or our SMTP setup, or our daemons' code, or... Assuming that the mod_perl app is the front-end for users and that you're trying to optimize for speed of responses, you should just use DProf to tell you which subroutines are using the most wall clock time. (I think it's dprofpp -t or something. Check the man page.) If the sub that uses IMAP is the slowest, then work on speeding up your IMAP server or the way you access it. CPU utilization may not be all that telling, since database stuff often takes the longest but doesn't burn much CPU. - Perrin
Re: Comparison of different caching schemes
On Sun, Dec 16, 2001 at 09:58:09AM +1100, Jeremy Howard wrote: Can anyone suggest a way (under Linux 2.4, if it's OS dependent) to get a log of CPU (and also IO preferably) usage by process name over some period of time? What about BSD Process Accounting (supported in most *nix systems) and sysstat (iostat + sar)? regards, Luciano Rocha -- Luciano Rocha, [EMAIL PROTECTED] The trouble with computers is that they do what you tell them, not what you want. -- D. Cohen
Re: Comparison of different caching schemes
On Sat, Dec 15, 2001 at 08:57:30PM -0500, Perrin Harkins wrote: One place that Rob and I still haven't found a good solution for profiling is trying to work out whether we should be focussing on optimising our mod_perl code, or our IMAP config, or our MySQL DB, or our SMTP setup, or our daemons' code, or... Assuming that the mod_perl app is the front-end for users and that you're trying to optimize for speed of responses, you should just use DProf to tell you which subroutines are using the most wall clock time. (I think it's dprofpp -t or something. Check the man page.) If the sub that uses IMAP is the slowest, then work on speeding up your IMAP server or the way you access it. CPU utilization may not be all that telling, since database stuff often takes the longest but doesn't burn much CPU. I agree 100%, wall clock time is the best metric. Consider that most apps are not CPU bound, they are I/O or network bound. Wall clock time measures that exactly. Another useful tool is truss/strace. Start up your server single process (-X) and then attach to it with strace. The -c and -r flags to strace are quite handy.. For example here's the cumulative stats for 'ls' % time seconds usecs/call callserrors syscall -- --- --- - - 52.290.004508 751 6 read 19.480.001679 560 3 write 12.930.001115 5620 close 6.010.000518 2323 old_mmap 5.450.000470 2122 2 open 1.220.000105 21 5 munmap 0.930.80 40 2 getdents64 0.710.61 321 fstat64 0.290.25 4 7 brk 0.240.21 11 2 mprotect 0.190.16 8 2 ioctl 0.120.10 10 1 uname 0.090.08 3 3 2 rt_sigaction 0.060.05 5 1 fcntl64 -- --- --- - - 100.000.008621 118 4 total -- Paul Lindner [EMAIL PROTECTED]| | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Re: Comparison of different caching schemes
I was using Cache::SharedMemoryCache on my system. I figured, Hey, it's RAM, right? It's gonna be WAY faster than anything disk-based. The thing you were missing is that on an OS with an aggressively caching filesystem (like Linux), frequently read files will end up cached in RAM anyway. The kernel can usually do a better job of managing an efficient cache than your program can. For what it's worth, DeWitt Clinton accompanied his first release of File::Cache (the precursor to Cache::FileCache) with a benchmark showing this same thing. That was the reason File::Cache was created. And ++ on Paul's comments about Devel::DProf and other profilers. - Perrin
RE: Comparison of different caching schemes
Another powerful tool for tracking down performance problems is perl's
profiler combined with Devel::DProf and Apache::DProf. Devel::DProf
is bundled with perl. Apache::DProf is hidden in the Apache-DB package
on CPAN.
Ya know the place in my original comment where I was optimizing a different
subsystem? I just discovered Devel::DProf last week (after 5 *years* of
perl... smacks forehead), and was using that. *AND* had improved a sore
spot's performance by 10% without even working hard, because of profiling.
Point taken.
At the same time I added some code to track the time it takes to
process a request using Time::HiRes. This value is set as a note via
$r-note('REQTIME'). A customlog directive takes care of dumping that
value in the logs...
Hmm... I was already logging a status message via warn(), I did the SAME
TRICK but stored it in a local variable because I didn't need to go as far
as a customlog...
Sounds like great minds think alike! :-)
L8r,
Rob
#!/usr/bin/perl -w
use Disclaimer qw/:standard/;
Re: Comparison of different caching schemes
On Fri, 14 Dec 2001, Perrin Harkins wrote: The thing you were missing is that on an OS with an aggressively caching filesystem (like Linux), frequently read files will end up cached in RAM anyway. The kernel can usually do a better job of managing an efficient cache than your program can. Plus for some reason IPC overhead seems to seriously degrade as the size of the overall shared memory segment increases. I'm not sure why this is but I think the docs for IPC::Shareable mention this. Maybe for _very_ small amounts of data, shared memory might still be a win. -dave /*== www.urth.org We await the New Sun ==*/
Re: Comparison of different caching schemes
So our solution was caching in-process with just a hash, and using a
DBI/mysql persistent store.
in pseudo code
sub get_stuff {
if (! $cache{$whatever} ) {
if !( $cache{whatever} = dbi_lookup()) {
$cache{$whatever}=derive_data_from_original_source($whatever);
dbi_save($cache_whatever);
}
}
$cache{$whatever}
}
That's actually a bit different. That would fail to notice updates between
processes until the in-memory cache was cleared. Still very useful for
read-only data or data that can be out of sync for some period though.
The filesystem based / time sensitive aging is a nice little thing we
should
put up in CPAN. We've just not done so yet.
How does it differ from the other solutions like Cache::FileCache? Is it
something you could add to an existing module?
- Perrin
Re: Comparison of different caching schemes
On December 14, 2001 03:04 pm, Perrin Harkins wrote:
So our solution was caching in-process with just a hash, and using a
DBI/mysql persistent store.
in pseudo code
sub get_stuff {
if (! $cache{$whatever} ) {
if !( $cache{whatever} = dbi_lookup()) {
$cache{$whatever}=derive_data_from_original_source($whatever);
dbi_save($cache_whatever);
}
}
$cache{$whatever}
}
That's actually a bit different. That would fail to notice updates between
processes until the in-memory cache was cleared. Still very useful for
read-only data or data that can be out of sync for some period though.
The filesystem based / time sensitive aging is a nice little thing we
should
put up in CPAN. We've just not done so yet.
How does it differ from the other solutions like Cache::FileCache? Is it
something you could add to an existing module?
Performance,and key structure, semantics. Its mostly different, not
better/worse.
=pod
=head1 NAME
UFMEDIA::CacheOneFile - cache a scalar value in a file on disk
=head1 SYNOPSIS
use UFMEDIA::CacheOneFile;
my $cache = new UFMEDIA::CacheOneFile(
cache_file = '/var/cache/myapp/flurge.cache',
max_age= 30,
refill_sub = sub { recalculate_flurges(31337, 'blue', 42) },
);
my $value = $cache-get_value;
=head1 DESCRIPTION
UFMEDIA::CacheOneFile enables you to cache a single scalar value in a file
on disk. Given a filename under a writable directory, a maximum age, and
a reference to a refill subroutine, a cache object will cache the result
of the refill subroutine in the file the first time Bget_value() is called,
and use the cached value for subsequent calls to Bget_value() until
Bmax_age the cach
e file is more than Bmax_age seconds old.
If multiple processes share a single cache file, the first process that
reads the cache file after it has expired will take responsibility for
replacing it with an up-to-date copy. Other processes needing up-to-date
information will wait for this to finish and will then use the new value.
=head1 AUTHOR
Mike Lyons [EMAIL PROTECTED]
=head1 SEE ALSO
perl(1).
=cut
- Perrin
--
Jay yohimbe Thorne [EMAIL PROTECTED]
Mgr Sys Tech, Userfriendly.org
Re: Comparison of different caching schemes
At 6:04 PM -0500 12/14/01, Perrin Harkins wrote: That's actually a bit different. That would fail to notice updates between processes until the in-memory cache was cleared. Still very useful for read-only data or data that can be out of sync for some period though. The primary problem with everything mentioned thus far is that they are almost entirely based around the concept of a single server. Caching schemes that are both fast and work across multiple servers and process instances are very hard to find. After reading the eToys article, I decided that BerkeleyDB was worth a look. We discovered that it was very fast and would make a great cache for some of our commonly used database queries. The problem was that we are running on 5 different servers, load balanced by Big/IP. Again, taking my que from the eToys article, I began working on a data exchange system (using perl instead of C) which multicasts data packets to the listening servers. So far, our only problems have been with the deadlocking (solved with db_deadlock), and a few corrupt records. I'm considering uploading to CPAN the stuff I've written for the caching, but haven't had the time to make it generic enough. I also haven't performed any benchmarks other than the Wow, that's a lot faster benchmark. One limitation to the stuff I've written is that the daemons (listeners) are non threaded, non forking. That and it's all based on UDP. Rob -- When I used a Mac, they laughed because I had no command prompt. When I used Linux, they laughed because I had no GUI.
Re: Comparison of different caching schemes
On December 14, 2001 03:53 pm, Robert Landrum wrote: At 6:04 PM -0500 12/14/01, Perrin Harkins wrote: That's actually a bit different. That would fail to notice updates between processes until the in-memory cache was cleared. Still very useful for read-only data or data that can be out of sync for some period though. The primary problem with everything mentioned thus far is that they are almost entirely based around the concept of a single server. Caching schemes that are both fast and work across multiple servers and process instances are very hard to find. After reading the eToys article, I decided that BerkeleyDB was worth a look. We discovered that it was very fast and would make a great cache for some of our commonly used database queries. The problem was that we are running on 5 different servers, load balanced by Big/IP. Again, taking my que from the eToys article, I began working on a data exchange system (using perl instead of C) which multicasts data packets to the listening servers. So far, our only problems have been with the deadlocking (solved with db_deadlock), and a few corrupt records. We did stuff based on kind of an Acceptable differences concept. Using round robinn dns, with predominantly winXX clients, you tend to get the same IP address from one request to tthe next. Changing, IIRC, on the dns cache timeout, from the soa record. With a bigIP or something, you can guarantee this user locality. So that leaves you with *appearing* to be in sync for the one local machine. Using a database with common connections from multiple machines or the one writer db machine, many read db machines concept, using replication or what have you for updates, you can present a pretty consistent database view, with a maximum difference on the read side of the replication time. We found that the database concept, plus using a local cache with a timeout of 5 seconds not only gave us a huge performance increase, but it gave us a maximum of about 6 seconds of data difference between machines. And since your average human viewer hits pages no faster than once every 20 seconds or so, it was pretty much invisible to the individual user. Of course search engines will be faster than this, but does being consistent for search engines matter all that much? I'm considering uploading to CPAN the stuff I've written for the caching, but haven't had the time to make it generic enough. I also haven't performed any benchmarks other than the Wow, that's a lot faster benchmark. One limitation to the stuff I've written is that the daemons (listeners) are non threaded, non forking. That and it's all based on UDP. Rob -- Jay yohimbe Thorne [EMAIL PROTECTED] Mgr Sys Tech, Userfriendly.org
Re: Comparison of different caching schemes
The thing you were missing is that on an OS with an aggressively caching filesystem (like Linux), frequently read files will end up cached in RAM anyway. The kernel can usually do a better job of managing an efficient cache than your program can. For what it's worth, DeWitt Clinton accompanied his first release of File::Cache (the precursor to Cache::FileCache) with a benchmark showing this same thing. That was the reason File::Cache was created. While that's true, there are still some problems with a file cache. Namely to get reasonably performance you have to do the directory hashing structure so that you don't end up with too many files (one for each key) in one directory. Thus for every add to the cache you have to: * stat each directory in the hash path and create it if it doesn't exist * open and create the file and write to it, close the file A similar method is required for reading. All that still takes a bit of time. This is where having some shared memory representation can be a really help since you don't have to traverse, open, read/write, close a file every time. Witness the performance of IPC::MM which seems to be mostly limited by the performance of the Storable module. I'm planning on doing another test which just stores some data without the 'streaming' to see which examples are really limited by Storable and which by their implementation, this might be useful for some people. And ++ on Paul's comments about Devel::DProf and other profilers. Ditto again. I've been using Apache::DProf recently and it's been great at tracking down exactly where time is spent in my program. If you have any performance problems, definitely use it first before making any assumptions. One question though, I have a call tree that looks a bit like: main - calls f1 - calls f2 - calls f3 - calls f2 The two calls to f2 may take completely different times. Using 'dprofpp -I', I can see what percentage of overall time is spent in 'f1 and children', 'f3 and children' and 'f2 and children'. But I can't see an easy way to tell 'time in f2 and children when f2 was called from f1' (or ditto for f3). Does that make sense? Rob
Re: Comparison of different caching schemes
Some more points. I'd like to point out that I don't think the lack of actual concurrency testing is a real problem, at leastfor most single CPU installations. If most of the time is spent doing other stuff in a request (which is most likely the case), then on average when a process goes to access the cache, nothing else will be accessing it, so it won't have to block to wait to get a lock. In that case, one process doing 10*N accesses is the same as N processes doing 10 accesses, and the results are still meaningful. Perrin Harkins pointed out IPC::MM which I've added to the test code. It's based on the MM library (http://www.engelschall.com/sw/mm/mm-1.1.3.tar.gz) and includes a hash and btree tied hash implementation. I've tried the tied hash and it performs extremely well. It seems to be limited mostly by the speed of the Storable module. Package C0 - In process hashSets per sec = 181181Gets per sec = 138242Mixes per sec = 128501Package C1 - Storable freeze/thawSets per sec = 2856Gets per sec = 7079Mixes per sec = 3728Package C2 - Cache::MmapSets per sec = 810Gets per sec = 2956Mixes per sec = 1185Package C3 - Cache::FileCacheSets per sec = 392Gets per sec = 813Mixes per sec = 496Package C4 - DBI with freeze/thawSets per sec = 660Gets per sec = 1923Mixes per sec = 885Package C5 - DBI (use updates with dup) with freeze/thawSets per sec = 676Gets per sec = 1866Mixes per sec = 943Package C6 - MLDBM::Sync::SDBM_FileSets per sec = 340Gets per sec = 1425Mixes per sec = 510Package C7 - Cache::SharedMemoryCacheSets per sec = 31Gets per sec = 21Mixes per sec = 24Package C8 - IPC::MMSets per sec = 2267Gets per sec = 5435Mixes per sec = 2769 Rob
Re: Comparison of different caching schemes
On Wed Dec 12, 2001 at 03:05:33PM +1100, Rob Mueller (fastmail) wrote: I tried out the following systems. * Null reference case (just store in 'in process' hash) * Storable reference case (just store in 'in process' hash after 'freeze') * Cache::Mmap (uses Storable) * Cache::FileCache (uses Storable) * DBI (I used InnoDB), use Storable, always do 'delete' then 'insert' * DBI, use Storable, do 'select' then 'insert' or 'update' * MLDBM::Sync::SDBM_File (uses Storable) * Cache::SharedMemoryCache (uses Storable) [...] Have I missed something obvious? Hi Rob, Maybe you can add my under construction Apache::Cache module to you test suite ? It is availlable on CPAN. Best regards -- ___ O l i v i e rP o i t r e y
Re: Comparison of different caching schemes
On Wed, Dec 12, 2001 at 03:05:33PM +1100, Rob Mueller (fastmail) wrote: I sat down the other day and wrote a test script to try out various caching implementations. The script is pretty basic at the moment, I just wanted to get an idea of the performance of different methods. Rob, wow! This is fantastic work! I'm wondering if you could include a test on the Cache::FileBackend class. The Cache::FileCache class is not optimized for the particular type of gets and sets that you were testing for. In fact, each of those requests will still need to go through the process of checking for object expiration, which may add the overhead that you observe in the benchmark. Not that I'm expecting Cache::FileBackend to do significantly better -- it still does name hashing, directory hashing, taint checking, cloning of objects, etc. But I am curious to see what difference it makes. You could try Cache::SharedMemoryBackend as well. In general the Cache::* modules were designed for clarity and ease of use in mind. For example, the modules tend to require absolutely no set-up work on the end user's part and try to be as fail-safe as possible. Thus there is run-time overhead involved. That said, I'm certainly not against performance. :) These benchmarks are going to be tremendously useful in identifying bottlenecks. However, I won't be able to optimize for these particular benchmarks, as Cache::Cache is designed to do something different than straight gets and sets. Again, thank you, Rob. This is great, -DeWitt
Re: Comparison of different caching schemes
Perrin Harkins wrote: Also, I'd like to see MLDBM + BerkeleyDB (not DB_File) with BerkeleyDB doing automatic locking, and IPC::MM, and IPC::Shareable, and IPC::ShareLite (though it doesn't serialize complex data by itself), and MySQL with standard tables. Of course I could just do them myself, since you were kind enough to provide code. IPC::ShareLite freezes/thaws the whole data structure, rather than just the hash element being accessed, IIRC, so is probably going to have extremely poor scaling characteristics. Worth adding to check, of course. MLDBM+BDB without using MLDBM::Sync is a cool idea that I hadn't thought of. That will certainly be interesting. Another interesting option is mapping a MySQL table data structure directly to the data structure being stored. This can be faster because you don't have to serialise, and also don't have to update the whole structure, just the fields that change (which is common in session handling and similar). Unfortunately this is hard to include in this benchmark, because the benchmark creates random complex structures. Maybe we'll create a custom one for this option for the sake of reference, even although it won't be directly comparable. I'm not sure what a 'standard table' in MySQL is any more... Berkeley, MyISAM, ISAM... I guess we can try all these, but that's benchmarking the DB rather than the caching scheme, and we're not about to try every DB server we can find!
Re: Comparison of different caching schemes
IPC::ShareLite freezes/thaws the whole data structure, rather than just the hash element being accessed, IIRC, so is probably going to have extremely poor scaling characteristics. Worth adding to check, of course. No, it's probably not worth it. It would be worth adding IPC::Shareable though, because people never believe me when I tell them to use something else. Having some numbers would help. Another interesting option is mapping a MySQL table data structure directly to the data structure being stored. That could be useful as part of a comparison for storing non-complex data, i.e. a single scalar value. I'm not sure what a 'standard table' in MySQL is any more... Berkeley, MyISAM, ISAM... I guess we can try all these, but that's benchmarking the DB rather than the caching scheme, and we're not about to try every DB server we can find! No, of course not. It may be that the performance characteristics of these table types are well known already and I just don't follow the MySQL scene well enough to know. I thought maybe the default tables type (MyISAM?) which doesn't support transactions would have better speed for dirt simple storage like this. - Perrin
Re: Comparison of different caching schemes
In general the Cache::* modules were designed for clarity and ease of use in mind. For example, the modules tend to require absolutely no set-up work on the end user's part and try to be as fail-safe as possible. Thus there is run-time overhead involved. That said, I'm certainly not against performance. :) These benchmarks are going to be tremendously useful in identifying bottlenecks. However, I won't be able to optimize for these particular benchmarks, as Cache::Cache is designed to do something different than straight gets and sets. Again, thank you, Rob. This is great, That's a good point. I probably should have added the features that each one can do to help with decisions. Cache::Cache does have the most options with regard to limiting time/size in the cache, so that could be a be factor in someones choice. * Cache::Mmap (uses Storable) - Can indirectly specify the maximum cache size, though purges are uneven depending on how well data hashes into different buckets - Has callback ability on a read/purge so you can move any purged data to a different data store if you want, and automatically retrieve it on next retrieve when it's not in the cache * Cache::FileCache (uses Storable) * Cache::SharedMemoryCache (uses Storable) - Can specify the maximum cache size (Cache::SizeAwareFileCache) and/or maximum time an object is allowed in the cache - Follows the Cache::Cache interface system * DBI (I used InnoDB), use Storable, always do 'delete' then 'insert' * DBI, use Storable, do 'select' then 'insert' or 'update' - Can't specifiy any limits directly - Could add a 'size' and 'timestamp' column to each row and use a daemon to iterate through and cleanup based on time and size * MLDBM::Sync::SDBM_File (uses Storable) * IPC::MM - Can't specifiy any limits directly - Could create secondary tied db/mm hash with key - [ size, timestamp ] mapping and use daemon to iterate through and cleanup based on time and size Rob
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 6 Dec 2001, Paul Lindner wrote: BTW -- I think where the docs are cached should be configurable. I don't like the idea of the document root writable by the web process. That's the price you pay for this functionality. Because we use Apache's native file serving code we need a url-directory mapping somewhere. uh, why couldn't Apache::CacheContent just set $r-filename(/where/we/put/the/cache/$file) ? If you add Bill's suggestion about caching on args, headers and whatnot you would (on some filesystems) need something like that anyway to make a hashed directory tree. - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do(); more than a billion impressions per week, http://valueclick.com
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Tue, Dec 11, 2001 at 01:50:52AM -0800, Ask Bjoern Hansen wrote: On Thu, 6 Dec 2001, Paul Lindner wrote: BTW -- I think where the docs are cached should be configurable. I don't like the idea of the document root writable by the web process. That's the price you pay for this functionality. Because we use Apache's native file serving code we need a url-directory mapping somewhere. uh, why couldn't Apache::CacheContent just set $r-filename(/where/we/put/the/cache/$file) ? Simplicity really. This was an example in our upcoming book so I didn't want to add a filename generator to the code, instead we use Apache's url-file mapping mechanism. Also this code was derived from a 404 error handler that I wrote ages ago :) I assume (since you suggested it) that you can set $r-filename to any file in any directory without adding a Directory config? I'll have to see how this interacts with the built-in access control logic . If you add Bill's suggestion about caching on args, headers and whatnot you would (on some filesystems) need something like that anyway to make a hashed directory tree. Right. A more elaborate Apache::CacheContent would have a filename hash function, and a separate cache directory structure along the lines of Cache::FileCache. I suppose that one could put the whole uri-cachefile mapping into a custom PerlTransHandler and leave Apache::CacheContent as-is.. -- Paul Lindner [EMAIL PROTECTED]| | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Tue, Dec 11, 2001 at 02:31:36AM -0800, Paul Lindner wrote: Right. A more elaborate Apache::CacheContent would have a filename hash function, and a separate cache directory structure along the lines of Cache::FileCache. Just curious -- any reason not to use Cache::Cache as the persistance mechanism? It was designed for exactly this scenario and could provide a nice abstraction for the filesystem or shared memory, as well as handle things like filename hashing and branching directories (and namespaces, size limits, OS independance, taint checking, and more). -DeWitt
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
Paul Lindner wrote: [snip] I suppose that one could put the whole uri-cachefile mapping into a custom PerlTransHandler and leave Apache::CacheContent as-is.. yeah, I think that we're starting to talk about two different approaches now. the cool thing about the current logic is that no filename mapping has to take place making it rather fast - basically, after a simple call to some cached stat() properties and you're done, Apache's native translation mechanism has done all the work. the price you pay for that quick simplicity is stuff written to your document root. adding a URI-filename translation step adds overhead, though it may be preferable to some. it shouldn't be a requirement, however. one of the neat things is about this module is that it makes (pretty creative) use of method handlers. the base class comes with disk_cache(), but memory_cache(), uri_cache(), i_dont_want_the_file_in_my_docroot_cache(), or whatever can be added either to the module proper or a subclass. so, maybe disk_cache() needs a better and less generic name, like docroot_cache(). --Geoff
Comparison of different caching schemes
Just thought people might be interested... I sat down the other day and wrote a test script to try out various caching implementations. The script is pretty basic at the moment, I just wanted to get an idea of the performance of different methods. The basic scenario is the common mod_perl situation: * Multiple processes * You need to share perl structures between processes * You want to index the data structure on some key * Same key is written and read multiple times I tried out the following systems. * Null reference case (just store in 'in process' hash) * Storable reference case (just store in 'in process' hash after 'freeze') * Cache::Mmap (uses Storable) * Cache::FileCache (uses Storable) * DBI (I used InnoDB), use Storable, always do 'delete' then 'insert' * DBI, use Storable, do 'select' then 'insert' or 'update' * MLDBM::Sync::SDBM_File (uses Storable) * Cache::SharedMemoryCache (uses Storable) For systems like Cache::* which can automatically delete items after an amount of time or size, I left these options off or made the cache big enough that this wouldn't happen. I've included the script for people to try out if they like and add other test cases. Now to the results, here they are. Package C0 - In process hashSets per sec = 147116Gets per sec = 81597Mixes per sec = 124120Package C1 - Storable freeze/thawSets per sec = 2665Gets per sec = 6653Mixes per sec = 3880Package C2 - Cache::MmapSets per sec = 809Gets per sec = 3235Mixes per sec = 1261Package C3 - Cache::FileCacheSets per sec = 393Gets per sec = 831Mixes per sec = 401Package C4 - DBI with freeze/thawSets per sec = 651Gets per sec = 1648Mixes per sec = 816Package C5 - DBI (use updates with dup) with freeze/thawSets per sec = 657Gets per sec = 1994Mixes per sec = 944Package C6 - MLDBM::Sync::SDBM_FileSets per sec = 334Gets per sec = 1279Mixes per sec = 524Package C7 - Cache::SharedMemoryCacheSets per sec = 42Gets per sec = 29Mixes per sec = 32 Notes: * System =Pentium III 866, Linux 2.4.16-0.6, Ext3 (no special file filesystem flags), MySQL (with InnoDB tables) * Null reference hash is slower reading because it does a very basic check to see that the retrieved hash has the same number of keys as the stored hash * Approximate performance order (best to worst) = Cache::Mmap, DBI, MLDBM::Sync::SDBM_File, Cache::FileCache, Cache::SharedMemoryCache * Rememberwhat Knuth said, "Premature optimisation is the root of all evil." This data won't help you if something else in your application is the bottleneck... * The code is available at: http://fastmail.fm/users/robm/perl/cacheperltest.pl Have I missed something obvious? Rob . - Grain of salt to be taken with this post
Re: Comparison of different caching schemes
I sat down the other day and wrote a test script to try out various caching implementations. Very interesting. Looks like Cache::Mmap deserves more attention (and maybe a Cache::Cache subclass). Have I missed something obvious? Nothing much, but I'd like to see how these numbers vary with the size of the data being written. And it would be good if there were a way to ensure that the data had not been corrupted at the end of a run. Also, I'd like to see MLDBM + BerkeleyDB (not DB_File) with BerkeleyDB doing automatic locking, and IPC::MM, and IPC::Shareable, and IPC::ShareLite (though it doesn't serialize complex data by itself), and MySQL with standard tables. Of course I could just do them myself, since you were kind enough to provide code. - Perrin
Re: Comparison of different caching schemes
Just wanted to add an extra thought that I forgot to include in the previous post. One important aspect missing from my tests is the actual concurrency testing. In mostreal world programs, multiple applications will be reading from/writing to the cache at the same time. Depending on the cache synchronisation scheme, you'll get varying levels of performance degradation: 1 being worst, 3 being the best. 1. Lock entire cache for a request(Cache::SharedMemoryCache, MySQL normal - table locks?) 2. Lock some part of cache for a request (Cache::Mmap buckets, MLDBM pages?) 3. Lock only the key/value for a request (Cache::FileCache, MySQL InnoDB - row locks?) Uggg, to do a complete test you really need to generate an entire modelling system: 1. Number of concurrent processes 2. Average reads/writes to cache per second 3. Ratio of reused/new entries etc Rob
Re: Comparison of different caching schemes
One important aspect missing from my tests is the actual concurrency testing. Oh, I guess I should have checked your code. I thought these were concurrent. That makes a huge difference. 2. Lock some part of cache for a request (Cache::Mmap buckets, MLDBM pages?) MLDBM::Sync locks the whole thing. - Perrin
[RFC] Apache::CacheContent - Caching PerlFixupHandler
Hi,
I would like to propose a new Apache module before I send it off to
CPAN. The name chosen is Apache::CacheContent.
It's pretty generic code, and is intended to be subclassed. It
handles the gory details of caching a page to disk and serving it up
until it expires.
It's derived from work done on the mod_perl Developer's Cookbook, so
it's already been reviewed by a number of people.
I've attached a README below. To download it go to
http://www.modperlcookbook.org/code.html
NAME
Apache::CacheContent - PerlFixupHandler class that caches dynamic
content
SYNOPSIS
* Make your method handler a subclass of Apache::CacheContent
* allow your web server process to write into portions of your document
root.
* Add a ttl() subroutine (optional)
* Add directives to your httpd.conf that are similar to these:
PerlModule MyHandler
# dynamic url
Location /dynamic
SetHandler perl-script
PerlHandler MyHandler-handler
/Location
# cached URL
Location /cached
SetHandler perl-script
PerlFixupHandler MyHandler-disk_cache
PerlSetVar CacheTTL 120 # in minutes...
/Location
DESCRIPTION
Note: This code is derived from the *Cookbook::CacheContent*
module, available as part of The mod_perl Developer's
Cookbook
The Apache::CacheContent module implements a PerlFixupHandler that helps
you to write handlers that can automatically cache generated web pages
to disk. This is a definite performance win for sites that end up
generating the exact same content for many users.
The module is written to use Apache's built-in file handling routines to
efficiently serve data to clients. This means that your code will not
need to worry about HTTP/1.X, byte ranges, if-modified-since, HEAD
requests, etc. It works by writing files into your DocumentRoot, so be
sure that your web server process can write there.
To use this you MUST use mod_perl method handlers. This means that your
version of mod_perl must support method handlers (the argument
EVERYTHING=1 to the mod_perl build will do this). Next you'll need to
have a content-generating mod_perl handler. If isn't a method handler
modify the *handler* subroutine to read:
sub handler ($$) {
my ($class, $r) = @_;
Next, make your handler a subclass of *Apache::CacheContent* by adding
an ISA entry:
@MyHandler::ISA = qw(Apache::CacheContent);
You may need to modify your handler code to only look at the *uri* of
the request. Remember, the cached content is independent of any query
string or form elements.
After this is done, you can activate your handler. To use your handler
in a fully dyamic way configure it as a PerlHandler in your httpd.conf,
like this:
PerlModule MyHandler
Location /dynamic
SetHandler perl-script
PerlHandler MyHandler-handler
/Location
So requests to *http://localhost/dynamic/foo.html* will call your
handler method directly. This is great for debugging and testing the
module. To activate the caching mechanism configure httpd.conf as
follows:
PerlModule MyHandler
Location /cached
SetHandler perl-script
PerlFixupHandler MyHandler-disk_cache
PerlSetVar CacheTTL 120 # in minutes..
/Location
Now when you access URLs like *http://localhost/cached/foo.html* the
content will be generated and stored in the file
*DocumentRoot*/cached/foo.html. Subsequent request for the same URL will
return the cached content, depending on the *CacheTTL* setting.
For further customization you can write your own *ttl* function that can
dynamically change the caching time based on the current request.
AUTHOR
Paul Lindner [EMAIL PROTECTED], Geoffrey Young, Randy Kobes
SEE ALSO
The example mod_perl method handler the CacheWeather manpage.
The mod_perl Developer's Cookbook
--
Paul Lindner [EMAIL PROTECTED]| | | | | | | | | |
mod_perl Developer's Cookbook http://www.modperlcookbook.org
Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
I would like to propose a new Apache module before I send it off to CPAN. The name chosen is Apache::CacheContent. This is very cool. I was planning to write one of these, and now I don't have to. Your implementation is short and interesting. I was planning to do it with a PerlFixupHandler and an Apache::Filter module to capture the output. While that approach wouldn't require the use of method handlers, I think yours may be easier for newbies because it doesn't require them to understand as many modules. The only real advantage of using Apache::Filter is that it would work well with existing Registry scripts. A couple of other C's for your R: A cache defines parameters that constitute a unique request. Your cache currently only handles the filename from the request as a parameter. It would be nice to also handle query args, POST data, and arbitrary headers like cookies or language choices. You could even support an optional request_keys method for handlers which would let people generate their own unique key based on their analysis of the request. Doing this would mean you would need to generate filenames based on the unique keys (probably by hashing, as in Cache::FileCache) and do an internal redirect to that file if available when someone sends a request that matches. Another thing that might be nice would be to store the TTL with the file rather than making the handler give it to you again each time. This is done in mod_proxy by putting an Expires header in the file and reading it before sending the file, but you could also store them in a dbm or something. Support for sending Expires headers automatically would also be useful. When I first thought about this problem, I wanted to do it the way Vignette StoryServer does: by having people link to the cached files directly and making the content generating code be the 404 handler for those files. That gives the best possible performance for cached files, since no PerlFixupHandler needs to run. The downside is that then you need an external process to go through and clean up expired files. It's also hard to handle complex cache criteria like query args. StoryServer does it by having really crazy generated file names and processing all the links to files on the way out so that they use the cached file names. Pretty ugly. I know you guys are pushing to get the book done, so don't feel pressured to address this stuff now. I think the current module looks more than good enough for an initial CPAN release. - Perrin
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
At 08:19 AM 12/06/01 -0800, Paul Lindner wrote: Ok, hit me over the head. Why wouldn't you want to use a caching proxy? BTW -- I think where the docs are cached should be configurable. I don't like the idea of the document root writable by the web process. Bill Moseley mailto:[EMAIL PROTECTED]
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 06 Dec 2001 10:04:26 -0800 Bill Moseley [EMAIL PROTECTED] wrote: BTW -- I think where the docs are cached should be configurable. I don't like the idea of the document root writable by the web process. Maybe: Alias /cached /tmp/cache -- Tatsuhiko Miyagawa [EMAIL PROTECTED]
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 6 Dec 2001 08:19:09 -0800
Paul Lindner [EMAIL PROTECTED] wrote:
I've attached a README below. To download it go to
http://www.modperlcookbook.org/code.html
Nice one. here's a patch to make the sample code work :)
--- CacheContent.pm~Thu Dec 6 22:11:35 2001
+++ CacheContent.pm Fri Dec 7 03:23:39 2001
@@ -6,6 +6,7 @@
@Apache::CacheContent::ISA = qw(Apache);
use Apache;
+use Apache::Log;
use Apache::Constants qw(OK SERVER_ERROR DECLINED);
use Apache::File ();
--- eg/CacheWeather.pm~ Thu Dec 6 08:10:09 2001
+++ eg/CacheWeather.pm Fri Dec 7 03:24:14 2001
@@ -8,7 +8,7 @@
use strict;
-@CacheWeather::ISA = qw(Cookbook::CacheContent);
+@CacheWeather::ISA = qw(Apache::CacheContent);
sub ttl {
my($self, $r) = @_;
--
Tatsuhiko Miyagawa [EMAIL PROTECTED]
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 6 Dec 2001, Paul Lindner wrote:
On Thu, Dec 06, 2001 at 10:04:26AM -0800, Bill Moseley wrote:
At 08:19 AM 12/06/01 -0800, Paul Lindner wrote:
Ok, hit me over the head. Why wouldn't you want to use a caching proxy?
Apache::CacheContent gives you more control over the caching process
and keeps the expiration headers from leaking to the browser. Or
maybe you want to dynamically control the TTL?
sub ttl {
...
if ($load_avg 5) {
return 60 * 5;
} else {
return 60;
}
}
While a ttl might be useful to some projects, others I'm sure would prefer
a per-hit checking, so you can say Yes, this thing has changed now.
Just a thought.
--
Matt/
/||** Founder and CTO ** ** http://axkit.com/ **
//||** AxKit.com Ltd ** ** XML Application Serving **
// ||** http://axkit.org ** ** XSLT, XPathScript, XSP **
// \\| // ** mod_perl news and resources: http://take23.org **
\\//
//\\
// \\
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
At 10:33 AM 12/06/01 -0800, Paul Lindner wrote: On Thu, Dec 06, 2001 at 10:04:26AM -0800, Bill Moseley wrote: At 08:19 AM 12/06/01 -0800, Paul Lindner wrote: Ok, hit me over the head. Why wouldn't you want to use a caching proxy? Apache::CacheContent gives you more control over the caching process and keeps the expiration headers from leaking to the browser. Ok, I see. Or maybe you want to dynamically control the TTL? Would you still use it with a front-end lightweight server? Even with caching, a mod_perl server is still used to send the cached file (possibly over 56K modem), right? Bill Moseley mailto:[EMAIL PROTECTED]
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, Dec 06, 2001 at 10:47:35AM -0800, Bill Moseley wrote: Ok, hit me over the head. Why wouldn't you want to use a caching proxy? Apache::CacheContent gives you more control over the caching process and keeps the expiration headers from leaking to the browser. Ok, I see. Or maybe you want to dynamically control the TTL? Would you still use it with a front-end lightweight server? Even with caching, a mod_perl server is still used to send the cached file (possibly over 56K modem), right? You definitely want a proxy-cache in front of your mod_perl server. One thing that I like about this module is that you can control the server-side caching of content separate from the client/browser cache. So, on to the RFC. Is the name acceptable for Apache::* I will endeavor to add any functionality that makes it worthy :) For example, I think adding a virtual method that generates the filename might be useful. -- Paul Lindner [EMAIL PROTECTED]| | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 6 Dec 2001, Paul Lindner wrote: On Thu, Dec 06, 2001 at 10:04:26AM -0800, Bill Moseley wrote: At 08:19 AM 12/06/01 -0800, Paul Lindner wrote: Ok, hit me over the head. Why wouldn't you want to use a caching proxy? Apache::CacheContent gives you more control over the caching process and keeps the expiration headers from leaking to the browser. Or maybe you want to dynamically control the TTL? You can use mod_accel to cache flexible backend: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.7.tar.gz mod_accel understands standard Expires and Cache-Control headers and special X-Accel-Expires header (it is not sent to client). Besides it allows to ignore Expires and Cache-Control headers from backend and set expiration by AccelDefaultExpire directive. Comparing to mod_proxy mod_accel reads backend response and closes connection to backend as soon as possible. There is no 2-second backend lingering close timeout with big answers and slow clients. Big answer means bigger then frontend kernel TCP-send buffer - 16K in FreeBSD and 64K in Linux by default. Besides mod_accel read whole POST body before connecting to backend. mod_accel can ignore client's Pragma: no-cache, Cache-Control: no-cache and even Authorization headers. mod_accel allow to not pass to backend some URLs. mod_accel allow to tune various buffer size and timeouts. mod_accel can cache responses with cookie-depended content. mod_accel can use busy locks and can limit number of connection to backend. mod_accel allows simple fault-tolerance with DNS-balanced backends. mod_accel logs various information about request processing. mod_accel can invalidate cache on per-URL basis. mod_accel has two drawbacks only - too much memory per connection (inherited Apache drawback) and Russian only documentation. Igor Sysoev
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
Hello, PLThat's the price you pay for this functionality. Because we use PLApache's native file serving code we need a url-directory mapping PLsomewhere. PL PLOf course you don't need to make the entire docroot writable, just the PLdirectory corresponding to your script. Apologies if this is obvious--I haven't downloaded and tried this module yet. But would it not be possible to specify a separate directory altogether and make it serveable (Directory ... ... Allow from all ...)? If so perhaps it'd be easy to add this as a configurable parameter. In general it is a fine idea to not make the DocumentRoot writeable by the web user. In fact, I believe it is a good policy that the web user should be able to write only to a small subset of controlled locations. Humbly, Andrew -- Andrew Ho http://www.tellme.com/ [EMAIL PROTECTED] Engineer [EMAIL PROTECTED] Voice 650-930-9062 Tellme Networks, Inc. 1-800-555-TELLFax 650-930-9101 --
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, Dec 06, 2001 at 12:55:25PM -0800, Andrew Ho wrote: Hello, PLThat's the price you pay for this functionality. Because we use PLApache's native file serving code we need a url-directory mapping PLsomewhere. PL PLOf course you don't need to make the entire docroot writable, just the PLdirectory corresponding to your script. Apologies if this is obvious--I haven't downloaded and tried this module yet. But would it not be possible to specify a separate directory altogether and make it serveable (Directory ... ... Allow from all ...)? If so perhaps it'd be easy to add this as a configurable parameter. Yes, you can do this using the regular Apache directives: # mkdir /var/cache/www/mydir # chown apache /var/cache/www/mydir # vi /etc/httpd/conf/httpd.conf Directory /var/cache/www/mydir AllowOverride None Order allow,deny Allow from all /Directory Alias /mydir/ /var/cache/www/mydir/ In general it is a fine idea to not make the DocumentRoot writeable by the web user. In fact, I believe it is a good policy that the web user should be able to write only to a small subset of controlled locations. Yes, I agree totally! I'll add some warning to the docs to make sure that people do not inadvertently misconfigure their servers.. -- Paul Lindner [EMAIL PROTECTED]| | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Apache::Registry caching of compiled scripts
Hi Does anybody know why Apache::Registry caches compiled scripts by their URI, instead of by their absolute path? At my site we have quite a lot of virtual hosts, which use the same base of cgi scripts. This causes the same scripts to be compiled and cached several times (one time for each virtual host). As I understand it, this behaviour COULD be changed by putting $Apache::Registry::NameWithVirtualHost = 0 in a perl start-up file, which would only cache on the relative URI (without the host name), but this won't really do, since not ALL the virtual hostst use the same base of CGI scripts. So I was just wondering why the compiled scripts are not cached under the absolute path to the script, instead of under the URI, as the former seems quite logical at first and would seem to solve some performance/memory issues when using virtual hosts. This design choice probably has a very good reason, which I am just unable to see. I would just like to find out in order to learn... :-) Nicholas Oxhøj
Re: Apache::Registry caching of compiled scripts
Nicholas Oxhj [EMAIL PROTECTED] said something to this effect on 10/11/2001: Does anybody know why Apache::Registry caches compiled scripts by their URI, instead of by their absolute path? Try Apache::RegistryNG, which caches scripts by their filename. (darren) -- Premature optimization is the root of all evil in programming. -- C.A.R. Hoare
Module that implements HTTP/1.1 caching?
Does anyone know if there is a perl module that implements HTTP/1.1 caching? I'd like to use one with Apache::ProxyRewrite and don't want to reinvent the wheel. :) Regards, Christian - Christian Gilmore Team Lead Web Infrastructure Tools IBM Software Group
Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
One of the shiny golden nuggets I received from said slice was a
shared memory cache. It was simple, it was elegant, it was
perfect. It was also based on IPC::Shareable. GREAT idea. BAD
juju.
Just use Cache::Cache. It's faster and easier.
Now, ya see...
Once upon a time, not many moons ago, the issue of Cache::Cache came up with
the SharedMemory Cache and the fact that it has NO locking semantics. When
I found this thread in searching for ways to implement my own locking scheme
to make up for this lack, I came upon YOUR comments that perhaps
Apache::Session::Lock::Semaphore could be used, without any of the rest of
the Apache::Session package.
That was a good enough lead for me.
So a went into the manpage, and I went into the module, and then I
mis-understood how the semaphore key was determined, and wasted a good hour
or two trying to patch it. Then I reverted to my BASICS: Data::Dumper is
your FRIEND. Print DEBUGGING messages. Duh, of course, except for some
reason I didn't think to worry about it, at first, in somebody else's
module. sigh So, see what I did wrong, undo the patches, and:
A:S:L:S makes the ASSUMPTION that the argument passed to its locking methods
is an Apache::Session object. Specifically, that it is a hashref of the
following (at least partial) structure:
{
data = {
_session_id = (something)
}
}
The _session_id is used as the seed for the locking semaphore. *IF* I
understood the requirements correctly, the _session_id has to be the same
FOR EVERY PROCESS in order for the locking to work as desired, for a given
shared data structure.
So my new caching code is at the end of this message.
***OH WOW!*** So, DURING the course of composing this message, I've
realized that the function expire_old_accounts() is now redundant!
Cache::Cache takes care of that, both with expires_in and max_size. I'm
leaving it in for reference, just to show how it's improved. :-)
***OH WOW! v1.1*** :-) I've also just now realized that the call to
bind_accounts() could actually go right inside lookup_account(), if:
1) lookup_account() is the only function using the cache, or
2) lookup_account() is ALWAYS THE FIRST function to access the cache, or
3) every OTHER function accessing the cache has the same call,
of the form bind() unless defined $to_bind;
I think for prudence I'll leave outside for now.
L8r,
Rob
%= snip =%
use Apache::Session::Lock::Semaphore ();
use Cache::SizeAwareSharedMemoryCache ();
# this is used in %cache_options, as well as for locking
use constant SIGNATURE = 'EXIT';
use constant MAX_ACCOUNTS = 300;
# use vars qw/%ACCOUNTS/;
use vars qw/$ACCOUNTS $locker/;
my %cache_options = ( namespace = SIGNATURE,
default_expires_in =
max_size = MAX_ACCOUNTS );
sub handler {
# ... init code here. parse $account from the request, and then:
bind_accounts() unless defined($ACCOUNTS);
# verify (access the cache)
my $accountinfo = lookup_account($account)
or $r-log_reason(no such account: $account), return
HTTP_NO_CONTENT;
# ... content here
}
# Bind the account variables to shared memory
sub bind_accounts {
warn bind_accounts: Binding shared memory if $debug;
$ACCOUNTS =
Cache::SizeAwareSharedMemoryCache-new( \%cache_options ) or
croak( Couldn't instantiate SizeAwareSharedMemoryCache : $! );
# Shut up Apache::Session::Lock::Semaphore
$ACCOUNTS-{data}-{_session_id} = join '', SIGNATURE, @INC;
$locker = Apache::Session::Lock::Semaphore-new();
# not quite ready to trust this yet. :-)
# We'll keep it separate for now.
#
#$ACCOUNTS-set('locker', $locker);
warn bind_accounts: done if $debug;
}
### DEPRECATED! Cache::Cache does this FOR us!
# bring the current session to the front and
# get rid of any that haven't been used recently
sub expire_old_accounts {
### DEPRECATED!
return;
my $id = shift;
warn expire_old_accounts: entered\n if $debug;
$locker-acquire_write_lock($ACCOUNTS);
#tied(%ACCOUNTS)-shlock;
my @accounts = grep( $id ne $_, @{$ACCOUNTS-get('QUEUE') || []} );
unshift @accounts, $id;
if (@accounts MAX_ACCOUNTS) {
my $to_delete = pop @accounts;
$ACCOUNTS-remove($to_delete);
}
$ACCOUNTS-set('QUEUE', \@accounts);
$locker-release_write_lock($ACCOUNTS);
#tied(%ACCOUNTS)-shunlock;
warn expire_old_accounts: done\n if $debug;
}
sub lookup_account {
my $id = shift;
warn lookup_account: begin if $debug;
expire_old_accounts($id);
warn lookup_account: Accessing \$ACCOUNTS{$id} if $debug;
my $s = $ACCOUNTS-get($id);
if (defined $s) {
# SUCCESSFUL CACHE HIT
warn lookup_account: Retrieved accountinfo from Cache (bypassing SQL) if
$debug;
return $s;
}
## NOT IN CACHE... refreshing.
warn lookup_account: preparing SQL if $debug;
# ... do some SQL here. Assign results
RE: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
The _session_id is used as the seed for the locking semaphore. *IF* I understood the requirements correctly, the _session_id has to be the same FOR EVERY PROCESS in order for the locking to work as desired, for a given shared data structure. Only if you want to lock the whole thing, rather than a single record. Cache::Cache typically updates just one record at a time, not the whole data structure, so you should only need to lock that one record. Uhh... good point, except that I don't trust the Cache code. The AUTHOR isn't ready to put his stamp of approval on the locking/updating. I'm running 10 hits/sec on this server, and last write wins, which ELIMINATES other writes, is not acceptable. I had a quick look at your code and it seems redundant with Cache::Cache. You're using the locking just to ensure safe updates, which is already done for you. Well, for a single, atomic lock, maybe. My two points above are the why of my hesitancy. Additionally, what if I decide to add to my handler? What if I update more than one thing at once? Now I've got the skeleton based on something that somebody trusts (A:S:L:S), vs what somebody thinks is alpha/beta (C:SASMC). In other words TIMTOWTDI! :-) L8r, Rob
RE: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
Uhh... good point, except that I don't trust the Cache code. The AUTHOR isn't ready to put his stamp of approval on the locking/updating. That sort of hesitancy is typical of CPAN. I wouldn't worry about it. I think I remember Randal saying he helped a bit with that part. In my opinion, there is no good reason to think that the Apache::Session locking code is in better shape than the Cache::Cache locking, unless you've personally reviewed the code in both modules. Well, the fact is, I respect your opinion. And YES, it seems like I'm doing more work than is probably necessary. I've been screwed over SO MANY TIMES by MYSELF not thinking of some little detail, than I've developed a tendency to design in redundant design redundancy :-) so that if one thing fails, the other will catch it. This reduces downtime... I'm running 10 hits/sec on this server, and last write wins, which ELIMINATES other writes, is not acceptable. As far as I can see, that's all that your code is doing. You're simply locking when you write, in order to prevent corruption. You aren't acquiring an exclusive lock when you read, so anyone could come in between your read and write and make an update which would get overwritten when you write, i.e. last write wins. Again, good point... I'm coding as if the WHOLE cache structure will break if any little thing gets out of line. I was trying to think in terms of data safety like one would with threading, because A) I was worried about weather shared memory was as sensitive to locks/corruption as threading, and B) I reviewed Apache::Session's lock code, but didn't review Cache::Cache's (20/20 hindsight, ya know). You're more than welcome to roll your own solution based on your personal preferences, but I don't want people to get the wrong idea about Cache::Cache. It handles the basic locking needed for safe updates. Then my code just got waaay simpler, both in terms of data flow and individual coding sections. THANK YOU! :-) L8r, Rob #!/usr/bin/perl -w use Disclaimer qw/:standard/;
Re: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
On Tue, Sep 04, 2001 at 12:14:52PM -0700, Rob Bloodgood wrote: ***OH WOW!*** So, DURING the course of composing this message, I've realized that the function expire_old_accounts() is now redundant! Cache::Cache takes care of that, both with expires_in and max_size. I'm leaving it in for reference, just to show how it's improved. :-) [snip] use Apache::Session::Lock::Semaphore (); use Cache::SizeAwareSharedMemoryCache (); # this is used in %cache_options, as well as for locking use constant SIGNATURE = 'EXIT'; use constant MAX_ACCOUNTS = 300; # use vars qw/%ACCOUNTS/; use vars qw/$ACCOUNTS $locker/; my %cache_options = ( namespace = SIGNATURE, default_expires_in = max_size = MAX_ACCOUNTS ); Very neat thought about how to use max_size to limit the the accounts! Unfortunately, you demonstrated that I did a *terrible* job at documenting what size means. It means size in bytes, not items. I will add max_items and limit_items to the TODO list. In the meantime, I will improve the documentation. -DeWitt
RE: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
What about my IPC::FsSharevars? I've once mentioned it on this list, but I don't have the time to read all list mail, so maybe I've missed some conclusions following the discussion from last time. I remember the post and went to find IPC::FsSharevars a while ago and was un-intrigued when I didn't find it on CPAN. has there been any feedback from the normal perl module forums? --Geoff
RE: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
At 20:37 Uhr -0400 4.9.2001, Geoffrey Young wrote: I remember the post and went to find IPC::FsSharevars a while ago and was un-intrigued when I didn't find it on CPAN. has there been any feedback from the normal perl module forums? I haven't announced it on other forums (yet). (I think it's more of a working version yet that needs feedback and some work to make it generally useable (i.e. under mod_perl). Which forum should I post on?) christian
Re: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
Christian == Christian Jaeger [EMAIL PROTECTED] writes: Christian I haven't announced it on other forums (yet). (I think it's Christian more of a working version yet that needs feedback and some Christian work to make it generally useable (i.e. under Christian mod_perl). Which forum should I post on?) If you put it on the CPAN with a version number below 1, that's usually a clue that it's still alpha or beta. Then you can announce it through the normal module announcement structures. If you hide it, I'm sure not installing it. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
Perrin == Perrin Harkins [EMAIL PROTECTED] writes: Uhh... good point, except that I don't trust the Cache code. The AUTHOR isn't ready to put his stamp of approval on the locking/updating. Perrin That sort of hesitancy is typical of CPAN. I wouldn't worry Perrin about it. I think I remember Randal saying he helped a bit Perrin with that part. I helped with the code that ensures that *file* writes are atomic updates. I taught DeWitt the trick of writing to a temp file, then renaming when ready, so that any readers see only the old file or the new file, but never a partially written file. I don't think Cache::Cache has enough logic for an atomic read-modify-write in any of its modes to implement (for example) a web hit counter. It has only atomic write. The last write wins strategy is fine for caching, but not for transacting, so I can see why Rob is a bit puzzled. It'd be nice if we could build a generic atomic read-modify-write, but now we're back to Apache::Session, which in spite of its name works fine away from Apache. :) Caching. An area of interest of mine, but I still don't seem to get around to really writing the framework I want, so all I can do is keep lobbing grenades into the parts I don't want. :) :) Sorry guys. :) -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)
I don't think Cache::Cache has enough logic for an atomic read-modify-write in any of its modes to implement (for example) a web hit counter. It has only atomic write. The last write wins strategy is fine for caching, but not for transacting, so I can see why Rob is a bit puzzled. In his example code he was only doing atomic writes as well, so it should work at least as well for his app as what he had before. It'd be nice if we could build a generic atomic read-modify-write Maybe a get_for_update() method is what's needed. It would block any other process from doing a set() or a get_for_update() until the set() for that key has completed. It's still just advisory locking though, so if you forget and use a regular get() for some data you later plan to set(), you will not be getting atomic read-modify-write. Maybe get() could be re-named to get_read_only(), or set a flag that prevents saving the fetched data. Most caching apps are happy with last save wins though, so I guess anything like that would need to be optional. - Perrin
Re: httpd.conf - caching
On Sun, 26 Aug 2001, allan wrote: im a bit confused or perhaps bewildered about finally having installed mod_perl on mac osX. Well, that's to be expected. It's a bit confusing first time through. these questions should apply cross platform 1) yesterday im quite sure my httpd.conf file was /usr/local/apache/conf/httpd.conf. today (after having played around) it suddenly appears to be /etc/httpd/httpd.conf You don't say how you played around ... sounds like you installed apache in two different locations. In any case, this isn't a mod_perl issue at all ... can this be? can i actually tell the sever somehow to use the one in /usr/local/apache/conf where the other conf files are. httpd -f /path/to/your/httpd.conf (Again, a very basic apache question. You really need to read the docs for apache. If you keep asking questions like this folks are going to quickly tire of helping you!) the only reason i discovered this was because i wanted to change the number of child processes to only one. which brings me to: 2) im experiencing some heavy modoule-caching annoyance i guess. if i run the simple hello.pm (see below) and make a very small change - the change might take effect immidiately in the browser but more frequently it will show a cached version of the page. (i noticed a pattern in my access_log that the last number-field on the lines would be the same if the page didnt change) Do you know why you are running mod_perl with your apache? The modoule-caching annoyance you speak of is the basic reason for the existence of mod_perl ... the idea is that the perl code in your module only gets compiled once rather than every time a user requests the URI. If you change the code in the module you won't see the changes until a new child process is spawned and compiles the new code (depending on your config maybe not even then ...) Most static sites don't need mod_perl ... if you are just installing apache for the first time maybe you should become familiar with running a webserver before you move on to mod_perlifying it ... ? i have read a little about this problem but dont understand the soloution(s). you can run httpd in single-process mode as you say, or you can look at Apache::Reload ... this forces your code to be recompiled every time it has changed (at the cost of checking out whether it has in fact changed, every request). Try perldoc Apache::Reload ... for instance what is the exact procedure for httpd -x? is it like this command: [localhost:/usr/local/apache/sbin] aju% httpd -x Well, actually it's httpd -X ... this command will do nothing on my system, not even complain. You should not see anything at the command line. The daemon is running, in single-process mode, still attached to the terminal in which you started it, waiting for a request. This again is very basic apache operation. Have you read the docs that come with your apache? if i manually configure my httpd.conf file to only one server-child i sometimes get two children? One parent, one child, probably. also there are some talk about global variables, but looking at the code i dont see any? There are quite a lot of talk about global variables, actually. It's a pretty big deal with this stuff. You need to go to the mod_perl guide, and spend _SEVERAL HOURS_ reading it and playing with your installation. If you devote the necessary time to educating yourself, you'll figure most of it out, and you can bring your tough questions to this list. allan Good luck, Nick
Re: httpd.conf - caching
nick thanks for the reply, and pointers - i guess i have completely lost overview thru the installation mist om mac osX. most of my questions were indeed more apache than mod_perl. Do you know why you are running mod_perl with your apache? to be honest, not entirely :-) The modoule-caching annoyance you speak of is the basic reason for the existence of mod_perl ... the idea is that the perl code in your module only gets compiled once rather than every time a user requests the URI. true, but if it *never* gets recompiled there are reasons to worry. There are quite a lot of talk about global variables, actually. It's a pretty big deal with this stuff. You need to go to the mod_perl guide, and spend _SEVERAL HOURS_ reading it and playing with your installation. If you devote the necessary time to educating yourself, you'll figure most of it out, and you can bring your tough questions to this list. well to me this is a tough situation and i have spend _SEVERAL HOURS_ reading it. anyway thanks again - i figure my real problem is, as you mention yourself, in understanding apache before mod_perl. allan
httpd.conf - random caching (2nd try)
im a bit confused or perhaps bewildered about finally having installed
mod_perl on mac osX.
these questions should apply cross platform
1)
yesterday im quite sure my httpd.conf file was /usr/local/apache/conf/httpd.conf.
today (after having played around) it suddenly appears to be /etc/httpd/httpd.conf
can this be?
and can i actually tell the server somehow to use the one in
/usr/local/apache/conf where the other conf files are.
the only reason i discovered this was because i wanted to change the
number of child processes to only one. which brings me to:
2)
im experiencing some heavy modoule-caching annoyance i guess. if i run the
simple hello.pm (see below) and make a very small change - the change
might take effect immediately in the browser but more frequently it will
show a random cached version of the page.
(i noticed a pattern in my access_log that the very last number-field on
each line would be the same if the page didnt change)
i have read a little about this problem in the online manual and the
writing apache modules book, but dont understand the soloution(s).
for instance what is the exact procedure for httpd -x?
is it like this command:
[localhost:/usr/local/apache/sbin] aju% httpd -x
this command will do nothing on my system, not even complain.
if i manually configure my httpd.conf file to only one server-child i
sometimes get two children?
also there are some talk about global variables, but looking at the code
i dont see any?
could there be other reeasons for this random caching?
thanks
allan
--hello.pm (from writing apache modules book)-
package Apache::Hello;
use strict;
use Apache::Constants qw(:common);
sub handler {
my $r = shift;
$r-content_type('text/html');
$r-send_http_header;
return OK if $r-header_only;
my $host = $r-get_remote_host;
$r-print(END);
HTML
HEADER
TITLEHello There/TITLE
/HEADER
BODY
H1Hello $host/H1
Who would take this book seriously if the first example didn't
say hello world?
/BODY
/HTML
END
return OK;
}
1;
__END__
Weird caching problem in windows browsers
hey guys i have a pretty big mod_perl system up and running and have run into a weird problem with browser caching (or rather, not caching) on windows. i have a series of complex forms that submit to and are generated by a mod_perl handler. each submitted form is validated and another form is generated. in some cases, there are about three or four forms in the sequence. for some reason, in windows, hitting the back button results in both IE and Netscape giving Warning: page has expired error pages. I have numerous places on the site that do this, and it's been working fine all this time. Now, the new forms all cause the error, whereas the old ones are still fine. I have been trying to figure out what's different between the old ones that work and the new ones but I can't really see anything. There are occasions with the newer ones where an entire sequence of forms submits to the same URL, like this: form method=post action=/path_to_my_handler/someNamedAction so I wrote some code to generate unique URLs for each subsequent POST and that still produces the same results. The weird thing is that if you refresh the page that produces the error, it will remember that page from then on and never show the error for that form again. Does anyone have any idea what might be causing this? I am stumped. It's also extra tough to debug because everything works perfectly in Linux so I have to run windows in VMware to even reproduce the error. Ugh. Kyle [EMAIL PROTECTED]
