RE: Dynamic content that is static
You might want to take a look at Strudel. It is a project people from my last job were working on: http://www.research.att.com/~mff/strudel/. Regards, Christian -Original Message- From: Philip Mak [mailto:[EMAIL PROTECTED]] Sent: Friday, December 22, 2000 8:09 PM To: [EMAIL PROTECTED] Subject: Dynamic content that is static Hi everyone, I have been going over the modperl tuning guide and the suggestions that people on this list sent me earlier. I've reduced MaxClients to 33 (each httpd process takes up 3-4% of my memory, so that's how much I can fit without swapping) so if the web server overloads again, at least it won't take the machine down with it. Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl. I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. (For example, http://www.animewallpapers.com/wallpapers/ccs.htm depends on header.asp, footer.asp, series.dat and index.inc. If none of those files change, the content of ccs.htm remains the same.) So, it would probably be more efficient if I had a /src directory and a /html directory. The /src directory could contain my modperl files and a Makefile that knows the dependencies; when I type "make", it will evaluate the modperl files and parse them into plain HTML files in the /html directory. Does anyone have any suggestions on how to implement this? Is there an existing tool for doing this? How can I evaluate modperl/Apache::ASP files from the command line? Thanks, -Philip Mak ([EMAIL PROTECTED])
Re: Dynamic content that is static
Apache::ASP has a cgi/asp script in the distribution that I use to generate apache-asp.org and chamas.com. Its a bit rough but works for static HTML generation from ASP scripts. Also you can consider using a combination of mod_proxy and specific headers like Expires to cache your content for X time like 1 hour. I have never done the latter, and you should use lwp-request -ed do see what headers your site is really sending, as ASP sends a no-cache pragma by default which you'll need to override, I forget which header at this time. The proxy method should work generally in this case, though I like generating content that can be offline. --Josh --- Philip Mak [EMAIL PROTECTED] wrote: Hi everyone, I have been going over the modperl tuning guide and the suggestions that people on this list sent me earlier. I've reduced MaxClients to 33 (each httpd process takes up 3-4% of my memory, so that's how much I can fit without swapping) so if the web server overloads again, at least it won't take the machine down with it. Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl. I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. (For example, http://www.animewallpapers.com/wallpapers/ccs.htm depends on header.asp, footer.asp, series.dat and index.inc. If none of those files change, the content of ccs.htm remains the same.) So, it would probably be more efficient if I had a /src directory and a /html directory. The /src directory could contain my modperl files and a Makefile that knows the dependencies; when I type "make", it will evaluate the modperl files and parse them into plain HTML files in the /html directory. Does anyone have any suggestions on how to implement this? Is there an existing tool for doing this? How can I evaluate modperl/Apache::ASP files from the command line? Thanks, -Philip Mak ([EMAIL PROTECTED]) __ Do You Yahoo!? Yahoo! Shopping - Thousands of Stores. Millions of Products. http://shopping.yahoo.com/
Re: Dynamic content that is static
On Fri, 22 Dec 2000, Philip Mak wrote: I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. (For example, http://www.animewallpapers.com/wallpapers/ccs.htm depends on header.asp, footer.asp, series.dat and index.inc. If none of those files change, the content of ccs.htm remains the same.) Thats just the sort of layout AxKit is great for. Its basically how Take23 works - even though it looks like a dynamically generated site, its not. Everything just comes from static files, and when those files change, AxKit recompiles the pages and caches the results (with some intelligence associated with the parameters used to determine if this is the same view of that URL). -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Dynamic content that is static
On Fri, Dec 22, 2000 at 09:51:55PM -0500, brian d foy wrote: however, i have been talking to a few people about something like a mod_makefile. :) I've used this approach succesfully on a lower volume site where the it was taking lots of time to build the final HTML but the data sources didn't change much. plugI have a module (Slay::Maker) I use for exactly this purpose that takes a "makefile" written in Perl and I use that to rebuild the pages, and if no page needs to be rebuilt, I can 304 the result/plug. A mod_makefile would be even nicer, being written in C. If you're looking for Perlish makes, Nick Ing-Simmons also has a Make.pm, there's a Makepp project out there, and plugI have an unreleased but releasable Make.pm that supports most of the GNU constructs/plug. Putting squid or something in front of a heavily trafficed site (and remembering to flush all or part of it's cache when you change the back end would definitely help) and using a makefile approach on the backend to avoid reading writing lots of rarely updated data every page view should both help. - Barrie
Dynamic content that is static
Hi everyone, I have been going over the modperl tuning guide and the suggestions that people on this list sent me earlier. I've reduced MaxClients to 33 (each httpd process takes up 3-4% of my memory, so that's how much I can fit without swapping) so if the web server overloads again, at least it won't take the machine down with it. Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl. I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. (For example, http://www.animewallpapers.com/wallpapers/ccs.htm depends on header.asp, footer.asp, series.dat and index.inc. If none of those files change, the content of ccs.htm remains the same.) So, it would probably be more efficient if I had a /src directory and a /html directory. The /src directory could contain my modperl files and a Makefile that knows the dependencies; when I type "make", it will evaluate the modperl files and parse them into plain HTML files in the /html directory. Does anyone have any suggestions on how to implement this? Is there an existing tool for doing this? How can I evaluate modperl/Apache::ASP files from the command line? Thanks, -Philip Mak ([EMAIL PROTECTED])
Re: Dynamic content that is static
Hi there, On Fri, 22 Dec 2000, Philip Mak wrote: I realized something, though: Although the pages on my site are dynamically generated, they are really static. You're not alone. Does anyone have any suggestions on how to implement this? Is there an existing tool for doing this? How can I evaluate modperl/Apache::ASP files from the command line? You could use 'lynx -source URI filename'. 73, Ged.
Re: Dynamic content that is static
Not necessarily. You can use mod_proxy to cache the dynamically generated pages on the lightweight apache. Check out http://perl.apache.org/guide/strategy.html#Apache_s_mod_proxy for details on what headers you'll need to set for caching to work. On Fri, 22 Dec 2000, Philip Mak wrote: Hi everyone, I have been going over the modperl tuning guide and the suggestions that people on this list sent me earlier. I've reduced MaxClients to 33 (each httpd process takes up 3-4% of my memory, so that's how much I can fit without swapping) so if the web server overloads again, at least it won't take the machine down with it. Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl.
Re: Dynamic content that is static
On Fri, 22 Dec 2000, Edward Moon wrote: Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl. Not necessarily. You can use mod_proxy to cache the dynamically generated pages on the lightweight apache. I thought about this... but I'm not sure how I would tell the lightweight Apache to refresh its cache when a file gets changed. I suppose I could graceful restart it, but the other webmasters of the site do not have root access. (Or is there another way? Is it possible to teach Apache or Squid that ccs.htm depends on header.asp, footer.asp, series.dat and index.inc?) Also, does this mess up the REMOTE_HOST variable, or is Apache smart enough to replace that with X-Forwarded-For when the forwarded traffic is being sent from a local priviledged process? -Philip Mak ([EMAIL PROTECTED])
Re: Dynamic content that is static
On Fri, 22 Dec 2000, Philip Mak wrote: So, it would probably be more efficient if I had a /src directory and a /html directory. The /src directory could contain my modperl files and a Makefile that knows the dependencies; when I type "make", it will evaluate the modperl files and parse them into plain HTML files in the /html directory. i've had great success with squid in http accelerator mode. we squeezed a factor of 100 in speed with just that. :) however, i have been talking to a few people about something like a mod_makefile. :) -- brian d foy [EMAIL PROTECTED] Perl hacker for hire
Re: Dynamic content that is static
I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. (For example, http://www.animewallpapers.com/wallpapers/ccs.htm depends on header.asp, footer.asp, series.dat and index.inc. If none of those files change, the content of ccs.htm remains the same.) You might want to consider using a tool other than mod_perl in this situation. There are preprocessor/compiler-type such as htmlpp or WML (both written in Perl), or you can build the pages in PHP (e.g.) and compile them into static pages with the command-line version. I've used both htmlpp and PHP, and both work well, and I drive them both with make. I don't know if either Mason or Embperl offer static compilation, but Mason has caching and I believe that Embperl is getting caching. AxKit is also very cool, and caches. -- Dave Seidel [EMAIL PROTECTED]
Re: Dynamic content that is static
On 22 Dec 2000, Dave Seidel wrote: I don't know if either Mason or Embperl offer static compilation, but Mason has caching and I believe that Embperl is getting caching.AxKit is also very cool, and caches. Using Mason to generate a set of HTML pages would not be too terribly difficult. If someone is interested in doing this and needs some guidance I'd be happy to help. It would be nice to include an offline site builder with Mason (or as a separate project). -dave /*== www.urth.org We await the New Sun ==*/
Re: Dynamic content that is static
I may be out of my realm here. I use mostly perl for everything and have
done similar things. Create a directory tree with the source files. In the
source files use something like %%INCLUDE_HEADER%% for each part of the page
that changes and have the script use flat text files for the build. Have
the script traverse the tree of the source files writing the output to the
html directory. Whenver you update the flat text files, just run the script
from the command line or write it to run from the web with a password.
Mason does something similar I believe.
You can even have the script write in the %%INCLUDES%% dynamically if you
take in the input and assign it like so.
$$Var = $value; instead of $input{'key'} = $value;
Then do the substitutions like so.
foreach (@variables){
$template_txt =~ s/%%$_%%/$$_/gi;
}
Works great for me. You can then make any change to the source.html page
and the flat text file without having to change the script.
John Michael
Not a mod perl solution, but it will work.
- Original Message -
From: "Dave Rolsky" [EMAIL PROTECTED]
To: "Dave Seidel" [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Friday, December 22, 2000 9:28 PM
Subject: Re: Dynamic content that is static
On 22 Dec 2000, Dave Seidel wrote:
I don't know if either Mason or Embperl offer static compilation, but
Mason has
caching and I believe that Embperl is getting caching. AxKit is also
very
cool, and caches.
Using Mason to generate a set of HTML pages would not be too terribly
difficult.
If someone is interested in doing this and needs some guidance I'd be
happy to help. It would be nice to include an offline site builder with
Mason (or as a separate project).
-dave
/*==
www.urth.org
We await the New Sun
==*/
Re: Dynamic content that is static
You should check out the documentation on mod_proxy to see what it's capable of: http://httpd.apache.org/docs/mod/mod_proxy.html You can specify expiration values and be assured that cached files older than expiry will be deleted. So, for example, if you know that your content gets updated every 48 hours you can specify 'CacheMaxExpire 48' and force the proxy server to retrieve a new copy every 48 hours. You can also set headers within a dynamic document that specifies an expiration time. Check out the link in my previous e-mail for more info. On Fri, 22 Dec 2000, Philip Mak wrote: I thought about this... but I'm not sure how I would tell the lightweight Apache to refresh its cache when a file gets changed. I suppose I could graceful restart it, but the other webmasters of the site do not have root access. (Or is there another way? Is it possible to teach Apache or Squid that ccs.htm depends on header.asp, footer.asp, series.dat and index.inc?) Also, does this mess up the REMOTE_HOST variable, or is Apache smart enough to replace that with X-Forwarded-For when the forwarded traffic is being sent from a local priviledged process? -Philip Mak ([EMAIL PROTECTED])
Re: Dynamic content that is static
At 09:08 PM 12/22/00 -0500, Philip Mak wrote: I realized something, though: Although the pages on my site are dynamically generated, they are really static. Their content doesn't change unless I change the files on the website. This doesn't really help with your ASP files, but have you looked at ttree in the Template Toolkit distribution? The problem, AFAIK, is that ttree only looks only at the top level documents and not included templates. I started to look at Template::Provider to see if there was an easy way to write out dependency information to a file, and then stat all those files every five minutes from a cron job and if anything changes, touch the top level files and then run ttree again. I'd like this because I'm generating cobarnded pages with mod_perl, and many of the pages are really static content. Bill Moseley mailto:[EMAIL PROTECTED]
Re: Dynamic content that is static
On Fri, 22 Dec 2000, Philip Mak wrote: Running a non-modperl apache that proxies to a modperl apache doesn't seem like it would help much because the vast majority of pages served require modperl. Not necessarily. You can use mod_proxy to cache the dynamically generated pages on the lightweight apache. I thought about this... but I'm not sure how I would tell the lightweight Apache to refresh its cache when a file gets changed. I suppose I could graceful restart it, but the other webmasters of the site do not have root access. (Or is there another way? Is it possible to teach Apache or Squid that ccs.htm depends on header.asp, footer.asp, series.dat and index.inc?) I don't know with Apache::ASP, but it can probably be done. With HTMl::Embperl it would be pretty trivial. You could probably get quite a gain with having a squid or a mod_proxy process in front. Both because they would slurp up the data and feed it to slow clients and because if you could cache the documents for just a minute or so it might save quite some hits on the backend. Also, does this mess up the REMOTE_HOST variable, or is Apache smart enough to replace that with X-Forwarded-For when the forwarded traffic is being sent from a local priviledged process? Have a look at ftp://ftp.netcetera.dk/pub/apache/mod_proxy_add_forward.c - ask -- ask bjoern hansen - http://ask.netcetera.dk/ more than 70M impressions per day, http://valueclick.com
