Re: caching proposals: SQLTransformer and Request

[David Kavanagh]

Christian,
When we've had the need to cache a query, we just throw the result (as a 
DOM object) into the sesssion. I'm including the sample pipeline. I'll 
typicailly aggregate the results of this with something else (usually a 
dynamic query) and I'm all set. When I want to clear the cache, the 
session attribute can be cleared.
(note, the getUserIdAction is something we wrote to grab the logged in 
user out of the session. We use container based authentication, but 
stash an object in the session for use in the sitemap).

We also ended up writing an action to invalidate a sesion object (used 
to clear the cache).

   
   
 
   
   
   
   
   
   

   
   
   
   
   
   
   
   
  
   
   
   
   

   
   
   
 
   
David

Christian Kurz wrote:

I just skimmed through the mailing list to find ideas of how to cache a
pipeline starting with a request generator and later on passing data
through the SQLTransformer.
Did you or anybody else follow up on this idea?

NB: Caching of the request generator would probably also need to cache
request parameters passed not part as part of the URL. As usually only
some request parameters are used in the pipeline, the sitemap element
might list the request parameters to consider when generating cache key:

   
   
   ...
-

A couple of things I'd like to do with Cocoon caching; let me know if
this
is crazy.
1. Add caching to the request generator.  Many of my pipelines are
transformations based upon the request, and since requestGenerator
currently
does not support caching, it means those transformations are always
re-done
(and often there is some sql at the end of the pipe which is really
slow).
I'd like to hash (or MD5?) the request string and use that as the cache
key
so that if I get the same one the pipeline knows it can skip over
everything.
2. Add caching to the SqlTransformer.  I know this sounds weird, but
hear me
out.  Our database is modified infrequently, so usually returns the same
data.  There is a datestamp in a special table which indicates when the
last
time the database was updated.  The SQL Transformer would remember the
time
of the last query.  I would have a new parameter to the sql transformer
to
indicate when the data is dirty.
   
 
   
LastUpdateTime looks like:

20021005144321 (Or whatever the xml date format is, I
forget).
SQLGenerator would resolve cocoon:/lastUpdateTime.xml.

I would then have a pipeline for lastUpdateTime.xml which would build it
by
querying my special table (but if you wanted, you could use some other
mechanism to build it).
SQLGenerator would compare the two dates and re-run the sql if it needed
to.
How does this sound?

Steve





-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
 



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: caching proposals: SQLTransformer and Request

[Christian Kurz]

I just skimmed through the mailing list to find ideas of how to cache a
pipeline starting with a request generator and later on passing data
through the SQLTransformer.

Did you or anybody else follow up on this idea?


NB: Caching of the request generator would probably also need to cache
request parameters passed not part as part of the URL. As usually only
some request parameters are used in the pipeline, the sitemap element
might list the request parameters to consider when generating cache key:




...

-

A couple of things I'd like to do with Cocoon caching; let me know if
this
is crazy.

1. Add caching to the request generator.  Many of my pipelines are
transformations based upon the request, and since requestGenerator
currently
does not support caching, it means those transformations are always
re-done
(and often there is some sql at the end of the pipe which is really
slow).
I'd like to hash (or MD5?) the request string and use that as the cache
key
so that if I get the same one the pipeline knows it can skip over
everything.

2. Add caching to the SqlTransformer.  I know this sounds weird, but
hear me
out.  Our database is modified infrequently, so usually returns the same
data.  There is a datestamp in a special table which indicates when the
last
time the database was updated.  The SQL Transformer would remember the
time
of the last query.  I would have a new parameter to the sql transformer
to
indicate when the data is dirty.


  


LastUpdateTime looks like:

20021005144321 (Or whatever the xml date format is, I
forget).

SQLGenerator would resolve cocoon:/lastUpdateTime.xml.

I would then have a pipeline for lastUpdateTime.xml which would build it
by
querying my special table (but if you wanted, you could use some other
mechanism to build it).

SQLGenerator would compare the two dates and re-run the sql if it needed
to.

How does this sound?

Steve





-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: caching proposals: SQLTransformer and Request

[Stephen Ng]

RequestGenerator *should* cache.  Give me a good reason why it shouldn't
have caching!

--Steve

> -Original Message-
> From: Stephen Ng [mailto:[EMAIL PROTECTED]]
> Sent: Friday, May 10, 2002 7:23 PM
> To: 'Vadim Gritsenko'; [EMAIL PROTECTED]
> Subject: RE: caching proposals: SQLTransformer and Request
>
>
> > May be you should consider different design, which is suited
> > better for
> > the problem? XSP pages with ESQL provide easy ability to program any
> > caching behavior.
> >
>
> What?!?  You just convinced me to go from XSP/ESQL to
> SQLTransformer!  The
> problem with ESQL is the Java recompilation, which is
> excessive when only
> the sql part is changing.
>
> Anyway, another way to solve my problem would be to simulate
> RequestGenerator using an XSP (have an xsp create the same
> format output as
> RequestGenerator).
>
> But this is really no different at all from rewriting
> RequestGenerator so it
> supports caching--the caching issues are exactly the same.
>
> So I think this means I should put caching into RequestGenerator.
>
> (As no-one has either protested or applauded, maybe not that
> many people are
> using RequestGenerator?)
>
> --Steve
>
>
> -
> Please check that your question has not already been answered in the
> FAQ before posting. <http://xml.apache.org/cocoon/faqs.html>
>
> To unsubscribe, e-mail: <[EMAIL PROTECTED]>
> For additional commands, e-mail: <[EMAIL PROTECTED]>
>


-
Please check that your question has not already been answered in the
FAQ before posting. <http://xml.apache.org/cocoon/faqs.html>

To unsubscribe, e-mail: <[EMAIL PROTECTED]>
For additional commands, e-mail: <[EMAIL PROTECTED]>

RE: caching proposals: SQLTransformer and Request

[Stephen Ng]

> May be you should consider different design, which is suited
> better for
> the problem? XSP pages with ESQL provide easy ability to program any
> caching behavior.
>

What?!?  You just convinced me to go from XSP/ESQL to SQLTransformer!  The
problem with ESQL is the Java recompilation, which is excessive when only
the sql part is changing.

Anyway, another way to solve my problem would be to simulate
RequestGenerator using an XSP (have an xsp create the same format output as
RequestGenerator).

But this is really no different at all from rewriting RequestGenerator so it
supports caching--the caching issues are exactly the same.

So I think this means I should put caching into RequestGenerator.

(As no-one has either protested or applauded, maybe not that many people are
using RequestGenerator?)

--Steve


-
Please check that your question has not already been answered in the
FAQ before posting. 

To unsubscribe, e-mail: <[EMAIL PROTECTED]>
For additional commands, e-mail: <[EMAIL PROTECTED]>

RE: caching proposals: SQLTransformer and Request

[Vadim Gritsenko]

> From: Stephen Ng [mailto:[EMAIL PROTECTED]]
> 
> A couple of things I'd like to do with Cocoon caching; let me know if
this
> is crazy.
> 
> 1. Add caching to the request generator.  Many of my pipelines are
> transformations based upon the request, and since requestGenerator
currently
> does not support caching, it means those transformations are always
re-done
> (and often there is some sql at the end of the pipe which is really
slow).
> I'd like to hash (or MD5?) the request string and use that as the
cache key
> so that if I get the same one the pipeline knows it can skip over
> everything.
> 
> 2. Add caching to the SqlTransformer.  I know this sounds weird, but
hear me
> out.  Our database is modified infrequently, so usually returns the
same
> data.  There is a datestamp in a special table which indicates when
the last
> time the database was updated.  The SQL Transformer would remember the
time
> of the last query.  I would have a new parameter to the sql
transformer to
> indicate when the data is dirty.
> 
> 
>value="cocoon:/lastUpdateTime.xml"/>
> 
> 
> LastUpdateTime looks like:
> 
> 20021005144321 (Or whatever the xml date format is, I
forget).
> 
> SQLGenerator would resolve cocoon:/lastUpdateTime.xml.
> 
> I would then have a pipeline for lastUpdateTime.xml which would build
it by
> querying my special table (but if you wanted, you could use some other
> mechanism to build it).
> 
> SQLGenerator would compare the two dates and re-run the sql if it
needed to.
> 
> How does this sound?

May be you should consider different design, which is suited better for
the problem? XSP pages with ESQL provide easy ability to program any
caching behavior.

Vadim

 
> Steve


-
Please check that your question has not already been answered in the
FAQ before posting. 

To unsubscribe, e-mail: <[EMAIL PROTECTED]>
For additional commands, e-mail: <[EMAIL PROTECTED]>