Re: [Radiant] Ideas for reimplementation of radiant caching
> Right. Caching the complete url in the database would help, > but would > not work in all cases, i.e. virtual pages that represent > multiple urls. > Perhaps we should try this and get some metrics? I was thinking that there would be a Page#calculate_urls method that would by default just return the result of Page#url but could be overridden to return multiple urls (in the case where a page knows all its urls) - otherwise there's the fallback to doing the tree search. I just did one quick metric - on my system, with ~900 pages, doing an update of the url cache for all pages 50 times took ~19 seconds - that's ~.4ms per page. Scaling up, that's 4 seconds if you modify a page with 10,000 children... I think that would be a reasonable response time... so the disadvantage at that end seems reasonable - just need to find out if the advantages are worth it. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Right. Caching the complete url in the database would help, but would
not work in all cases, i.e. virtual pages that represent multiple urls.
Perhaps we should try this and get some metrics?
Sean
Daniel Sheppard wrote:
> Maybe I need to dump my assumptions for a moment. Knowing the page that
> we're about to be rendering from the cache would be a useful thing. Why
> can't we do it? Because the performance of Page.find_by_url sucks.
>
> I talked a little bit about that suckage a while ago, and somebody
> (Adam? Sean?) suggested having a table to directly find lookup a page
> from it's url from the database. I was originally dismissive, but the
> idea is starting to grow on me.
>
> - Create a page_urls table
> - Whenever the slug or type of a page changes, that page and all of its
> children (recursively) would update the page_urls table with their
> current url(s).
>
> If a page isn't found using a direct lookup, a regular Page.find_by_url
> call would be made, so that we can still maintain all the flexibility
> that provides.
>
> Advantages of this system are:
>
> - faster to lookup pages by url (though I don't know if it will be fast
> enough that I'd want it before the cache check)
>
> - all children of a page could be discovered recursively through:
> PageUrl.find(:all, :include => :page, :conditions => 'url like ?',
> "#{parent_url}%")
>
> - old urls could be left in the table - automatically giving 'cool urls'
> - unless another page comes to replace it or the page is no longer
> published, a page is always available through it's old address. Old urls
> would probably get a marker and redirect to the real urls of a page.
>
>
> Disadvantages:
>
> - Database isn't third normal form (the url is a calculated field, not
> raw data), makes my inner database purist feel queasy.
>
> - Long save times when slug/type changes
>
>
>
> I should probably do some other things instead, but I might have a look
> at this.
>
>
>
> Dan.
> ___
> Radiant mailing list
> Post: Radiant@lists.radiantcms.org
> Search: http://radiantcms.org/mailing-list/search/
> Site: http://lists.radiantcms.org/mailman/listinfo/radiant
>
>
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Maybe I need to dump my assumptions for a moment. Knowing the page that
we're about to be rendering from the cache would be a useful thing. Why
can't we do it? Because the performance of Page.find_by_url sucks.
I talked a little bit about that suckage a while ago, and somebody
(Adam? Sean?) suggested having a table to directly find lookup a page
from it's url from the database. I was originally dismissive, but the
idea is starting to grow on me.
- Create a page_urls table
- Whenever the slug or type of a page changes, that page and all of its
children (recursively) would update the page_urls table with their
current url(s).
If a page isn't found using a direct lookup, a regular Page.find_by_url
call would be made, so that we can still maintain all the flexibility
that provides.
Advantages of this system are:
- faster to lookup pages by url (though I don't know if it will be fast
enough that I'd want it before the cache check)
- all children of a page could be discovered recursively through:
PageUrl.find(:all, :include => :page, :conditions => 'url like ?',
"#{parent_url}%")
- old urls could be left in the table - automatically giving 'cool urls'
- unless another page comes to replace it or the page is no longer
published, a page is always available through it's old address. Old urls
would probably get a marker and redirect to the real urls of a page.
Disadvantages:
- Database isn't third normal form (the url is a calculated field, not
raw data), makes my inner database purist feel queasy.
- Long save times when slug/type changes
I should probably do some other things instead, but I might have a look
at this.
Dan.
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
> H... does rails give you access to the session cookie before it
> gets sent off in the response? Perhaps the site_controller could be
> changed to clear out that cookie if the request is going to be cached
-
> giving a warning / raising an exception if the session is not empty.
> You'd still need a seperate layer of access control, that slots in
> before the cache handler, but if you could use rails sessions instead
> of rolling your own that would be useful. I'll have a little play.
Phew. Just had a look at how rails does sessions. Not very pretty -
session handling is actually done by CGI::Session.
It would probably be possible to get rid of session :off and get the
same behaviour with:
def show_uncached_page(url)
@page = find_page(url)
unless @page.nil?
@page.process(request, response)
if live? and @page.cache?
@cache.cache_response(url, response)
else
response.instance_eval { @cgi.instance_eval {
@output_cookies.clear }}
end
@performed_render = true
else
render :template => 'site/not_found', :status => 404
end
rescue Page::MissingRootPageError
redirect_to welcome_url
end
But that's really messy. Don't like it one bit.
I'd suggest instead:
class ProtectedSiteController < SiteController
def show_page
url = params[:url] = "/protected/#{params[:url]}"
if is_allowed(url)
super
else
# do something else
end
end
private
def is_allowed(url)
#test something
end
end
end
And in your extension:
define_routes do |map|
protected.connect '/protected/*url', :action => 'show_page',
:controller => 'protected'
end
Give it a go.
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
OK. Point taken. I was really just putting my two cents in on a
subject that's important to me personally... ie user variable content
and ACLs.
It would be nice to do something like (I haven't actually looked
at the code, my bad):
page = Page.new
page.headers
page.content = Cache.get_content('/cache/'+cache_file)
page.render
Anyway I'd better have a look at the code myself before I make
a/more of a fool of myself.
I wasn't trying to make a fool out of you - I completely understand what
you're trying to achieve, but asking the pages to handle authentication
isn't the way to go about it unless you're prepared to totally kill the
caching performance.
The way to implement this would probably be to introduce a filter to the
site controller that does ACL checks based on paths - check the ACL for
the path before doing anything else that site_controller does. Introduce
a new LoginPage type that is not cached that is responsible for setting
the authentication cookies. Rails doesn't give you as much control over
session creation as I'm used to in java servlets, so you'll have to take
over session control in your own code - You need to have your filter
check for the existance of a session, but not create one if it doesn't
exist.
H... does rails give you access to the session cookie before it gets
sent off in the response? Perhaps the site_controller could be changed
to clear out that cookie if the request is going to be cached - giving a
warning / raising an exception if the session is not empty. You'd still
need a seperate layer of access control, that slots in before the cache
handler, but if you could use rails sessions instead of rolling your own
that would be useful. I'll have a little play.
Dan.
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
I too have a need to implement a system that prevents displaying pages unless the user is authenticated. I will need levels of authentication and have more than just a handful of users. Smells like sessions to me but do I understand correctly that radiant's caching prevents this? I'm not even smart enough to know what would be required to make a user authentication system play nicely with radiant. Thoughts/suggestions? -Chris -- Posted via http://www.ruby-forum.com/. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
OK. Point taken. I was really just putting my two cents in on a
subject that's important to me personally... ie user variable content
and ACLs.
It would be nice to do something like (I haven't actually looked at
the code, my bad):
page = Page.new
page.headers
page.content = Cache.get_content('/cache/'+cache_file)
page.render
Anyway I'd better have a look at the code myself before I make a/more
of a fool of myself.
Cheers,
On 26/02/2007, at 1:57 PM, Daniel Sheppard wrote:
What I was suggesting wasn't that Page.find_by_url be called on
every page request The original email was about changes to the
caching mechanism...
My suggestion, working from your outline:
a) check for the existance of /cache/.yml
b) check for the validity of /cache/.yml
if file exists and is valid:
c) call the page.headers method to set the page headers
d) send /cache/.data
else
e) Call Page.find_by_url to find the page
f) call the page.headers method to set the page headers
g) call the page.render method to render the page content
There's a flaw in your logic - You can't call page.headers until
after you've called Page.find_by_url.
Dan.
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
What I was suggesting wasn't that Page.find_by_url be called on every page request The original email was about changes to the caching mechanism... My suggestion, working from your outline: a) check for the existance of /cache/.yml b) check for the validity of /cache/.yml if file exists and is valid: c) call the page.headers method to set the page headers d) send /cache/.data else e) Call Page.find_by_url to find the page f) call the page.headers method to set the page headers g) call the page.render method to render the page content There's a flaw in your logic - You can't call page.headers until after you've called Page.find_by_url. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
On 23/02/2007, at 1:51 PM, Daniel Sheppard wrote: to caching, isn't the major problem with implementing sessions in Radiant that the headers are cached as well? This seems to be over optimisation... Just cache the page text and let the headers be generated. Surely there isn't that large a hit in performance, and the advantage of using individual user sessions is great. Or make it an option at least, since most user ACL type systems require sessions... To serve a page currently, the following steps are performed: a) check for the existance of /cache/.yml b) check for the validity of /cache/.yml if file exists and is valid: c) send /cache/.data else d) Call Page.find_by_url to find the page e) call the page.headers method to set the page headers f) call the page.render method to render the page content Daniel, What I was suggesting wasn't that Page.find_by_url be called on every page request The original email was about changes to the caching mechanism... My suggestion, working from your outline: a) check for the existance of /cache/.yml b) check for the validity of /cache/.yml if file exists and is valid: c) call the page.headers method to set the page headers d) send /cache/.data else e) Call Page.find_by_url to find the page f) call the page.headers method to set the page headers g) call the page.render method to render the page content I was saying that the headers could be sent on every request. This allows ACLs to work for the main site pages... at the moment (as I understand it) the headers - so this includes session info - are cached and only update once per 5 minutes... This is unworkable if you want conditional content - ie if user registered show registered_user_page else show login_page I certainly wasn't saying that page caching wasn't needed, and can even see how powerful the radiant caching is since it essentially turns the site into static content, once cached. But caching headers doesn't seem workable in the long-term - or will need to be at least an option in the caching mechanism in order to implement more dynamic behaviour. Of course this then opens up the site to all kinds of dynamic content and caching by page url would not work... since two users could potentially have different pages with the same url... but it at least allows session data to be used on a page by page basis... Perhaps caching page parts would work? Anyway this is all stuff that will have to be worked out if blogging is going to be possible with Radiant... Cheers, Adam A quick experiment to see the performance impact of making a Page.find_by_url call on every request - using my current database containing a few hundred pages, I'll pick a random deep page. With the current caching mechanism, running: ab2 -n 1000 -c 2 'http://localhost/articles/2006/04/13/2006-melbourne-international- comed y-festival-reviews/levelland/' I ran it first using the current radiant caching mechanism, and then again modifying the site_controller to always do a Page.find_by_url regardless of whether the page is cached or not. Original cache : 160 requests / second Modified cache : 4 requests / second So, "Surely there isn't that large a hit in performance" unless two orders of magnitude doesn't constitute a large hit for you, I hope you see why the cache behaves as it does. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
> I know this is a little late (email from last month)... but > in regard > to caching, isn't the major problem with implementing sessions in > Radiant that the headers are cached as well? This seems to be over > optimisation... Just cache the page text and let the headers be > generated. Surely there isn't that large a hit in performance, and > the advantage of using individual user sessions is great. Or make it > an option at least, since most user ACL type systems require > sessions... To serve a page currently, the following steps are performed: a) check for the existance of /cache/.yml b) check for the validity of /cache/.yml if file exists and is valid: c) send /cache/.data else d) Call Page.find_by_url to find the page e) call the page.headers method to set the page headers f) call the page.render method to render the page content A quick experiment to see the performance impact of making a Page.find_by_url call on every request - using my current database containing a few hundred pages, I'll pick a random deep page. With the current caching mechanism, running: ab2 -n 1000 -c 2 'http://localhost/articles/2006/04/13/2006-melbourne-international-comed y-festival-reviews/levelland/' I ran it first using the current radiant caching mechanism, and then again modifying the site_controller to always do a Page.find_by_url regardless of whether the page is cached or not. Original cache : 160 requests / second Modified cache : 4 requests / second So, "Surely there isn't that large a hit in performance" unless two orders of magnitude doesn't constitute a large hit for you, I hope you see why the cache behaves as it does. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
I know this is a little late (email from last month)... but in regard to caching, isn't the major problem with implementing sessions in Radiant that the headers are cached as well? This seems to be over optimisation... Just cache the page text and let the headers be generated. Surely there isn't that large a hit in performance, and the advantage of using individual user sessions is great. Or make it an option at least, since most user ACL type systems require sessions... > > Daniel Sheppard wrote: > > >>From looking at response_cache, there are only two features > that it > > > implements that the default rails action caching didn't (easily): > > > > > > - Expire Everything > > > - Expire After time > > > > It also caches the entire response (headers + body). > > Yeah, I left that off because I thought that the rails action cache > already did that - but regardless, the action_cache plugin adds that > behaviour. I'll push on ahead. > ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Regarding to caching, I might just try the following: 1. Make the page-parts accessible via urls. ( /pagename/partname ) 2. Make the snippets accessible via urls.( /snippet/snippetname ... might clash ) 3. Have a method on PagePart which'll determine if their rendered context is dynamic or not. (I.e. if it depends on the clock, or on outer datasource then dynamic) 4. Let the controller set up an ssi object, give it the url ( Ssi.new url ), put it on stack 5.a When a part/snippet/layout called for rendering and it is dynamic ssi.append "", and push a new ssi on the stack with nil url (of course that stack can be the callstack but that's intursive) After the normal rendering write the popped new ssi to the public dir if it has an url 5.b else append the render to the top ssi object ( ssi.append context ), and give it a url if that's nil I expect to get an ssi file that includes in the dynamic parts, which made accesible via urls. Those dynamic radiant pageparts can in turn include in more static ssi files, and those can include in dynamic parts again... which isn't desirable if slow, but I'll fight that another day. This is all possible because apache2 can do ssi with our radiant output too (filter). I also expect this to be a rocket compared to what we have now. If I want to control the outgoing header I'd have to mark the outer page dynamic so at first it can have-a-go and do the filtering only with its output, but this is in the clouds, I can't see if this can be done or not. Apache might just hate us trying to set a header once he already sent out one because of the ssi. I'll have to try that. Overhead: little. Dynamic things push and pop once, nothing else. Static things do write things twice ( one for ssi and one for response ) but then in return it will not run again since the ssi will take over the load. Dependency: when we edit in admin, delete all corresponding ssi... which is not obvious a tiny bit, I should have left some breadcrumbs behind when appended the just edited part to an ssi. Those dependent ssis now need to be deleted. I wish I could do all this in an extension, but I'm so far unable to override anything in Page. If someone with a strong ruby-kungfu can help me... Until then I'll just extend it. Laszlo -- Posted via http://www.ruby-forum.com/. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
> It also seems to be a good idea not to put post request's > response into > a cache. > A post supposed to change data, and the outcome of that should not be > cached - I should not say OK if you just signed up as root... class Page def cache? request.get? end end Perhaps that should be the definition in core. Should fix that up. As for parameters - a workaround that also gives you prettier urls would be to have your page return itself from find_by_url and extract the params from the url, so /products?offset=1200&limit=10 becomes /products/perpage/10/page/120 If you're doing things with query strings, I generally wouldn't allow the page to be cached - there's too much variability - offset=1200&limit=10 or limit=10&offset=1200? You could also introduce a custom controller there, which might be a better approach for trying to introduce such functionality into a CMS. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Radiant currently caches only the page urls, without the params after
the ?
Am I the only one who thinks this is bad?
For example my products page listing works by
/products?offset=1200&limit=10
and I must turn off caching to make this work. Why not just let the
cache
work on the full uri? People do flip on a catalog back and forth, so I
did
benefit from changing this. If I had truly dynamic content, I'd turn off
the caching.
This in an extension did the trick:
class SiteController < ApplicationController
def show_page
response.headers.delete('Cache-Control')
uri = request.request_uri.to_s[1..-1]
url = params[:url].to_s
if live? and (@cache.response_cached?(uri))
@cache.update_response(uri, response)
@performed_render = true
else
show_uncached_page(url, uri)
end
end
private
def show_uncached_page(url, uri)
@page = find_page(url)
unless @page.nil?
@page.request=request
@page.process(request, response)
@cache.cache_response(uri, response) if live? and @page.cache? #and
not request.post?
@performed_render = true
else
render :template => 'site/not_found', :status => 404
end
rescue Page::MissingRootPageError
redirect_to welcome_url
end
end
It also seems to be a good idea not to put post request's response into
a cache.
A post supposed to change data, and the outcome of that should not be
cached - I should not say OK if you just signed up as root...
What do you think what will this change break?
Laszlo
--
Posted via http://www.ruby-forum.com/.
___
Radiant mailing list
Post: Radiant@lists.radiantcms.org
Search: http://radiantcms.org/mailing-list/search/
Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
> > Well then can we make Apache serve everything? Why not have the options > > to make Radiant generate a full directory of HTML files. A possible way > > to support having select pieces of the site be dynamic is to use Apache > > server side includes to make calls back to Radiant for specific pieces. > > This would be an ideal case, but I think the path Radiant takes is a good > compromise. The advantage of Radiant's caching system over static files is > that you can include headers. Sometimes this kind of information gets lost so here is a reminder: The Corex branch (an experimental transition from Trunk to Mental) has had a working implementation of a new caching mechanism that writes files to public/ once they are requested (GET). Since public/ has precedence over the application controller you can get extremely good performance results. Unfortunately, this type of static caching turned to out to be inflexible because headers cannot be cached. Otherwise it would have been a powerful alternative to the current caching mechanism. Too bad headers can't be reliably modified with tags. On 1/23/07, Sean Cribbs <[EMAIL PROTECTED]> wrote: > > > > > - It's annoying to have to clear page cache after every edit in order to > > view your change. > > The cache is automatically cleared for the page you edited, but only that > page. If you edit a layout or snippet, the whole cache is cleared. > > > Also, I see no reason why we can't attach a Preview button directly to > > each page edit screen. > > I think this would be a nice feature too. PDI? > > > Well then can we make Apache serve everything? Why not have the options > > to make Radiant generate a full directory of HTML files. A possible way > > to support having select pieces of the site be dynamic is to use Apache > > server side includes to make calls back to Radiant for specific pieces. > > This would be an ideal case, but I think the path Radiant takes is a good > compromise. The advantage of Radiant's caching system over static files is > that you can include headers. > > I know you're probably using Apache in a figurative sense, but not everyone > uses Apache. kckcc.edu runs quite speedily and effortlessly (excepting the > site map) on Litespeed. We use the built-in LSAPI Rails bridge, which is > purported to have a 30% boost over FastCGI. There are other ways to eek out > performance too, not all of which have to do with the caching. > > > Are there any other reasons for changing Radiant's caching? > > I don't see any. It's "good enough" for most cases, I believe. > > The unmentioned alternative, of course, is memcached. It would take minimal > changes to the code, probably a single line in environment.rb , since > ResponseCache piggy-backs on the Rails caching mechanism. However, not > everyone could run memcached. > > Sean > ___ > Radiant mailing list > Post: Radiant@lists.radiantcms.org > Search: http://radiantcms.org/mailing-list/search/ > Site: > http://lists.radiantcms.org/mailman/listinfo/radiant > > -- Alexander Horn http://www2.truman.edu/~ah428 . ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
I don't see any. It's "good enough" for most cases, I believe. The unmentioned alternative, of course, is memcached. It would take minimal changes to the code, probably a single line in environment.rb , since ResponseCache piggy-backs on the Rails caching mechanism. However, not everyone could run memcached. Actually, ResponseCache doesn't piggy-back on the rails caching mechanism - at least not in the sense where it would be simple to swap in one of the other rails caching backends. However, I think this might actually be the reason why ResponseCache manages to perform so well - all the abstraction in the rails caching creates a big performance hit. Writing an extension that uses memcache rather than the filesystem would require replacing the entire ResponseCache class. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
- It's annoying to have to clear page cache after every edit in order to view your change. The cache is automatically cleared for the page you edited, but only that page. If you edit a layout or snippet, the whole cache is cleared. Also, I see no reason why we can't attach a Preview button directly to each page edit screen. I think this would be a nice feature too. PDI? Well then can we make Apache serve everything? Why not have the options to make Radiant generate a full directory of HTML files. A possible way to support having select pieces of the site be dynamic is to use Apache server side includes to make calls back to Radiant for specific pieces. This would be an ideal case, but I think the path Radiant takes is a good compromise. The advantage of Radiant's caching system over static files is that you can include headers. I know you're probably using Apache in a figurative sense, but not everyone uses Apache. kckcc.edu runs quite speedily and effortlessly (excepting the site map) on Litespeed. We use the built-in LSAPI Rails bridge, which is purported to have a 30% boost over FastCGI. There are other ways to eek out performance too, not all of which have to do with the caching. Are there any other reasons for changing Radiant's caching? I don't see any. It's "good enough" for most cases, I believe. The unmentioned alternative, of course, is memcached. It would take minimal changes to the code, probably a single line in environment.rb, since ResponseCache piggy-backs on the Rails caching mechanism. However, not everyone could run memcached. Sean ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Before changing how the caching works, I would ask, what are the reasons for modifying caching. I can think of a few. - It's annoying to have to clear page cache after every edit in order to view your change. The solution Dan suggests would make it possible to know exactly which pages need their cache cleared. However, having a "versioning" system like BJ suggests would also help to accomplish this goal. And even currently, you can have a dev URL to your site where none of the pages are cached. Also, I see no reason why we can't attach a Preview button directly to each page edit screen. - Apache is faster than Radiant's cache. Well then can we make Apache serve everything? Why not have the options to make Radiant generate a full directory of HTML files. A possible way to support having select pieces of the site be dynamic is to use Apache server side includes to make calls back to Radiant for specific pieces. Are there any other reasons for changing Radiant's caching? -- Posted via http://www.ruby-forum.com/. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Jeepers. I just did the coding to use action_cache instead of the the ResponseCache mechanism, feeling all proud I decided to run some benchmarks. Raw apache can serve up a 60Kb page on my server at 677 requests/sec. Action cache turned out to be two orders of magnitude slower than that - 6.7req/sec - and that's using mod_xsendfile. Without, it's half that again - 3.2req/sec. Radiant's ResponseCache can spit out a 60Kb page on my server at 65 requests/sec. That's an order of magnitude slower than raw apache (677 req/sec), but still quite a resonable rate. I don't really see anything in action cache to account for such a performance hit - I'm guessing it's in the underlying rails fragment cache and the many levels of redirection there. Seems playing with the cache was a waste of time. Probably should have done my last benchmark first - the simplest rails app I could make serving up just the string 'hello' can serve only 73 requests/sec - If I had've just compared that to the ResponseCache speed I wouldn't have even bothered. I still want to get handling of if-modified-since and 304 responses into the ResponseCache, as well as some finer control over the cache timeout for a page on a page-by-page basis, but looks like action_cache is not the way to go there. Dan. > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Daniel Sheppard > Sent: Wednesday, 3 January 2007 2:52 PM > To: radiant@lists.radiantcms.org > Subject: Re: [Radiant] Ideas for reimplementation of radiant caching > > >From looking at response_cache, there are only two features that it > implements that the default rails action caching didn't (easily): > > - Expire Everything > - Expire After time > > Am I missing something else? If I'm not, then I think I might > have found > something that will give a big jump on what I want to get done: > > http://www.agilewebdevelopment.com/plugins/action_cache ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
> Daniel Sheppard wrote: > >>From looking at response_cache, there are only two features that it > > implements that the default rails action caching didn't (easily): > > > > - Expire Everything > > - Expire After time > > It also caches the entire response (headers + body). Yeah, I left that off because I thought that the rails action cache already did that - but regardless, the action_cache plugin adds that behaviour. I'll push on ahead. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Daniel Sheppard wrote: >>From looking at response_cache, there are only two features that it > implements that the default rails action caching didn't (easily): > > - Expire Everything > - Expire After time It also caches the entire response (headers + body). > Am I missing something else? If I'm not, then I think I might have found > something that will give a big jump on what I want to get done: > > http://www.agilewebdevelopment.com/plugins/action_cache That looks very nice. Please do investigate. -- John Long http://wiseheartdesign.com ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
>From looking at response_cache, there are only two features that it implements that the default rails action caching didn't (easily): - Expire Everything - Expire After time Am I missing something else? If I'm not, then I think I might have found something that will give a big jump on what I want to get done: http://www.agilewebdevelopment.com/plugins/action_cache This performs both of the above functions plus: - If-Modified-Since/Last-Modified headers (really important for any page with heavy traffic) - Expiry Time can easily be set on a page by page basis. - Optionally use X-Sendfile for making the webserver do the heavy lifting - All of the rails cache storage types (though only file-based will work with X-Sendfile - Regexp based expiry (great for expiring my asset transforms on an asset update) Get back to me asap if I'm missing some other magic feature of the response cache, otherwise I'll see how I go with ripping it out. Dan. ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Daniel Sheppard wrote: > I want radiant to have a more robust caching mechanism than the current > 'expire every x minutes' method of doing things - I'd like it to be able > to handle just removing things from the cache when things actually > change. I've roughed out a plan on how I think this should be done in a > safe manner. > > It sounds like it should work, but this is mainly just what I thought of > on the way to work this morning - I'd like people to poke holes in it > and figure out any edge-cases I'm missing out on - if this process EVER > misses out on the clearing of a cached page, it's pretty much useless. > > I see a bit off overhead being added to the rendering process (since > after_initialize hooks are a performance no-no), but since this should > reduce the need for rendering by a huge amount, I'm hoping it'll balance > out. > > I will probably put off an implementation of this until after I finish > off my asset extension and get other things running, but I might pull > this forward for asset caching as well... not sure yet. I not opposed to what you've written above, but wonder if there might be a simpler way that what you have proposed. I'm wondering if deleting the whole cache every time a change is made might work better. Currently, the cache is cleared when a snippet or layout is edited. This seems to be a bit time consuming as the cache is stored on disk, but justifiable in the case of snippets and layouts. To speed it up I'm wondering if we could get better performance just storing the rendered content in the db. Delete queries tend to be fast and Radiant is already serving the content in the first place (so there is no performance gain because the Web server can serve the page and Radiant doesn't have to). Whatever course we took we'd need to see the numbers, but I favor a simpler approach if performance is comparable. -- John Long http://wiseheartdesign.com ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Hi Dan, Ruben, Actually I once build a commercial Enterprise CMS where a custom content proxy would cache everything. It was flushed on time-out /and/ upon changes in the content. The proxy allowed large companies to set up caches wherever they pleased and still have a central repository. The caching was fairly complicated as we had a broad definition of what constituted as a change. The most difficult part is pages that contain generated stuff that uses the structure of the content tree, like menus. So my question to you is: how do you flush pages with menus that are generated from the content tree? Regards, Erik. Ruben D. Orduz wrote: > Just as an anecdote, I've worked with paid enterprise-grade CMS's, > open-source CMS's and I've only seen two types of caching: > > 1) The classic: time expiry unless forced by hand. > 2) Tag-level caching: You can choose which tags NOT to cache. So for > example, in a news site, pretty much the whole site could be cached, > except for content produced by <%breaking-news%>. > -- Erik van Oosten http://day-to-day-stuff.blogspot.com/ ___ Radiant mailing list Post: Radiant@lists.radiantcms.org Search: http://radiantcms.org/mailing-list/search/ Site: http://lists.radiantcms.org/mailman/listinfo/radiant
Re: [Radiant] Ideas for reimplementation of radiant caching
Dan,
This sounds like a pretty good system.
I'm not sure if my suggestions/concerns are directly related caching,
but they kind of go with it.
The biggest hole in Radiant that I can see is that there is not proper
version management and publishing capabilities.
I work for a large state government organization and I have to use
Microsoft CMS 8 hours a day. It does lots of stuff completely
terribly, but one nice thing about it is it's version
management/publishing system.
Here's what I mean:
You have an existing page, you change some content, there is no way to
preview it or approval process for it other than publishing it and
viewing it live. Maybe this is by choice (I hope not), but I could see
how this is a major reason for lack of adoption on a larger scale.
What I'd like to see:
After a page is changed, a new version is saved and the status for the
new version is set to draft. There's a way to preview this page, and
then publish/approve the page and it would then be live. This is when
it would be "cached".
This still allows for dynamic content and definitely takes Radiant to
the next level for me (dare I say, "enterprise ready").
My $.02, I don't know if you've considered it in your caching scheme.
- BJ Clark
On 1/1/07, Ruben D. Orduz <[EMAIL PROTECTED]> wrote:
> Dan,
>
> It seems you have given this quite a bit of thought, and it could
> work. I'm just worried about when will the ROI of performace hit make
> it be worth.
>
> Just as an anecdote, I've worked with paid enterprise-grade CMS's,
> open-source CMS's and I've only seen two types of caching:
>
> 1) The classic: time expiry unless forced by hand.
> 2) Tag-level caching: You can choose which tags NOT to cache. So for
> example, in a news site, pretty much the whole site could be cached,
> except for content produced by <%breaking-news%>.
>
> Maybe something like that could be adapted for Radiant (i.e. give the
> choice which caching is desired).
>
>
>
> On 1/1/07, Daniel Sheppard <[EMAIL PROTECTED]> wrote:
> > I want radiant to have a more robust caching mechanism than the current
> > 'expire every x minutes' method of doing things - I'd like it to be able
> > to handle just removing things from the cache when things actually
> > change. I've roughed out a plan on how I think this should be done in a
> > safe manner.
> >
> > It sounds like it should work, but this is mainly just what I thought of
> > on the way to work this morning - I'd like people to poke holes in it
> > and figure out any edge-cases I'm missing out on - if this process EVER
> > misses out on the clearing of a cached page, it's pretty much useless.
> >
> > I see a bit off overhead being added to the rendering process (since
> > after_initialize hooks are a performance no-no), but since this should
> > reduce the need for rendering by a huge amount, I'm hoping it'll balance
> > out.
> >
> > I will probably put off an implementation of this until after I finish
> > off my asset extension and get other things running, but I might pull
> > this forward for asset caching as well... not sure yet.
> >
> > --
> >
> > Example Site layout:
> >
> > Pages:
> > + Homepage ('main', 'sidebar')
> > + Articles ('main', 'sidebar')
> > - Article1 ('full', 'extract')
> > - Article2 ('full', 'extract')
> >
> > Snippets:
> > - Header
> > - Footer
> >
> > Layouts
> > - Default
> >
> >
> > Homepage:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 'default' layout:
> >
> >
> >
> >
> >
> > --
> >
> > In this example, the homepage needs to be re-rendered when:
> > - homepage is modified
> > - new article child is created
> > - article1/2 is updated/removed
> > - article1/2 has extract added/removed
> > - default layout is modified
> > - header snippet is modified
> > - footer snippet is modified
> > - (please tell me if I'm missing cases here)
> >
> > To know all those things, the following should be more than enough to
> > keep track of:
> >
> > - any page which is read from db during the render
> > - any snippet which is read from db during the render
> > - any layout which is read from db during the render
> >
> >
> >
> > New Table:
> >
> > create_table 'cache_dependencies' do |t|
> > t.column :page_id, :integer
> > t.column :depends_on, :integer
> > t.column :cache_type, :string
> > end
> >
> > Since this is just a scratch table, perhaps this should use something
> > like madeline or maybe just a tree of files that are locked on write
> > instead of being in the main database file.
> >
> > Filesystem cache:
> > pages/
> > page.data
> > page.cacheinfo
> >
> >
> > An after_save is added to each snippet/layout:
> >
> > CacheDependencies.find_all_by_cache_type_and_depends_on('sni
Re: [Radiant] Ideas for reimplementation of radiant caching
Dan,
It seems you have given this quite a bit of thought, and it could
work. I'm just worried about when will the ROI of performace hit make
it be worth.
Just as an anecdote, I've worked with paid enterprise-grade CMS's,
open-source CMS's and I've only seen two types of caching:
1) The classic: time expiry unless forced by hand.
2) Tag-level caching: You can choose which tags NOT to cache. So for
example, in a news site, pretty much the whole site could be cached,
except for content produced by <%breaking-news%>.
Maybe something like that could be adapted for Radiant (i.e. give the
choice which caching is desired).
On 1/1/07, Daniel Sheppard <[EMAIL PROTECTED]> wrote:
> I want radiant to have a more robust caching mechanism than the current
> 'expire every x minutes' method of doing things - I'd like it to be able
> to handle just removing things from the cache when things actually
> change. I've roughed out a plan on how I think this should be done in a
> safe manner.
>
> It sounds like it should work, but this is mainly just what I thought of
> on the way to work this morning - I'd like people to poke holes in it
> and figure out any edge-cases I'm missing out on - if this process EVER
> misses out on the clearing of a cached page, it's pretty much useless.
>
> I see a bit off overhead being added to the rendering process (since
> after_initialize hooks are a performance no-no), but since this should
> reduce the need for rendering by a huge amount, I'm hoping it'll balance
> out.
>
> I will probably put off an implementation of this until after I finish
> off my asset extension and get other things running, but I might pull
> this forward for asset caching as well... not sure yet.
>
> --
>
> Example Site layout:
>
> Pages:
> + Homepage ('main', 'sidebar')
> + Articles ('main', 'sidebar')
> - Article1 ('full', 'extract')
> - Article2 ('full', 'extract')
>
> Snippets:
> - Header
> - Footer
>
> Layouts
> - Default
>
>
> Homepage:
>
>
>
>
>
>
>
>
>
>
>
> 'default' layout:
>
>
>
>
>
> --
>
> In this example, the homepage needs to be re-rendered when:
> - homepage is modified
> - new article child is created
> - article1/2 is updated/removed
> - article1/2 has extract added/removed
> - default layout is modified
> - header snippet is modified
> - footer snippet is modified
> - (please tell me if I'm missing cases here)
>
> To know all those things, the following should be more than enough to
> keep track of:
>
> - any page which is read from db during the render
> - any snippet which is read from db during the render
> - any layout which is read from db during the render
>
>
>
> New Table:
>
> create_table 'cache_dependencies' do |t|
> t.column :page_id, :integer
> t.column :depends_on, :integer
> t.column :cache_type, :string
> end
>
> Since this is just a scratch table, perhaps this should use something
> like madeline or maybe just a tree of files that are locked on write
> instead of being in the main database file.
>
> Filesystem cache:
> pages/
> page.data
> page.cacheinfo
>
>
> An after_save is added to each snippet/layout:
>
> CacheDependencies.find_all_by_cache_type_and_depends_on('snippet',
> snippet.id) do |cd|
> if(File.exists("pages/#{cd.page_id}.cacheinfo"))
> File.rm("pages/#{cd.page_id}.cacheinfo")
> end
> end
>
> An after_save is added to each page to do the same, and also an
> after_create and after_destroy are added to page that will clear
> anything depending on the parent when a child is created or removed.
>
> ---
>
> A new class, CacheInfo is registered as an 'after_initialize' on Page,
> Layout and Snippet.
>
> When rendering begins, initialize the CacheInfo object:
> CacheInfo.reset
>
> When a page/layout/snippet is loaded, it's id is added to a hash in
> CacheInfo.
>
> When the rendering is complete, the cacheinfo is written to the
> cache_dependencies table, looking something like:
>
> CacheDependencies.transaction do
> CacheDependencies.delete_all(:conditions => ['page_id = ?',
> page.id])
> page_depends.each do |pd|
> CacheDependencies.create(:page_id => page.id, :depends_on =>
> pd, :cache_type => 'page')
> end
> snippet_depends.each do |pd|
> CacheDependencies.create(:page_id => page.id, :depends_on =>
> pd, :cache_type => 'snippet')
> end
> layout_depends.each do |pd|
> CacheDependencies.create(:page_id => page.id, :depends_on =>
> pd, :cache_type => 'layout')
> end
> end
>
>
> On each render, if the page.cacheinfo file exists, the page is loaded
> - if so, render the page from scratch
> - if not:
> - se
