[web2py] Re: Invalidated RAM cache from outside web2py to handle refreshing of asynchronous data
both memcache and redis are available as cache backends in web2py. Of
course memcache and redis are "external" piece of softwares but you can use
the same syntax in web2py that you use for cache.ram .
Speed difference between cache.ram and memcache or redis is negligible, as
long as the machine running redis or memcache is local or very close to the
machine running web2py.
Il giorno venerdì 11 gennaio 2013 15:56:15 UTC+1, Daniel Gonzalez ha
scritto:
>
> Thanks Niphlod. With that I can proceed to implementation.
>
> I am also considering another option: use memcached. This is not
> integrated with web2py (as far as I know), but would maybe simplify my
> implementation.
>
> Do you think the performance impact of using memcached (which is accessed
> via tcp) compared to using web2py's cache.ram (which lives in the web2py
> memory map) is very big?
>
> On Friday, January 11, 2013 3:06:40 PM UTC+1, Niphlod wrote:
>>
>> 1. if you use cache.ram you can't access the ram that your web2py process
>> is using from outside that process.
>>In that case you can code a controller that does the trick and
>> call it, e.g., with curl from the commandline.
>> 2. same thing, curl from the commandline, or use urllib2, or something
>> that basically does a request to your webserver.
>>
>> Il giorno venerdì 11 gennaio 2013 14:53:27 UTC+1, Daniel Gonzalez ha
>> scritto:
>>>
>>> Thanks, that makes sense. In that context, I have two questions:
>>>
>>>1. How do I call "cache.ram.clear" from outside web2py. What web2py
>>>modules should I include?
>>>2. How can I call a controller disguised as a user? I think this
>>>would be the most robust way, if it is possible ...
>>>
>>>
>>> On Friday, January 11, 2013 2:42:37 PM UTC+1, Niphlod wrote:
Cache is always "related" to a key. In your function, that key is
request.env.path_info
Let's assume your key is "*the_key*"
If you need to "invalidate" the cache, you should use cache.ram.clear("
*the_key*") or cache.ram("*the_key*", None). From here one, there are
2 paths ahead of you:
- you have an empty cache and it will be filled at the 1st request
coming from an user after you called clear()
- you want to fill that key after clear() ASAP
In the 1st case, after calling clear() you don't have to do nothing.
In the 2nd case, either you call the controller "disguised as a user"
or you process the data and store it using cache.ram("*the_key*",
thevalue)
Il giorno venerdì 11 gennaio 2013 13:07:08 UTC+1, Daniel Gonzalez ha
scritto:
>
> Hi,
>
> I am serving some data which is expensive to compute with
> a @service.jsonrpc controller. I want to cache the result in RAM, using
> something like:
>
> @cache(request.env.path_info,time_expire=1000,cache_model=cache.ram)
>
> My data will not change very often, so I want to have a big ttl
> (actually, I would like a ttl "forever", is this possible?)
>
> But my main problem is this: in my architecture, web2py does not know
> when new data is available (the data is in a third-party database,
> completely out of control of web2py). Changes to the data happen
> asynchronously, according to business processes.
>
> I have three pieces:
>
>1. web2py serving the data to the clients, via jsonrpc
>2. a small script which knows (long-polling) when changes in the
>third-party database occur
>3. a library to process the information in the third-party
>database. Calling this must be avoided at all costs, *except* when we
> have
>actual changes (which is why I want to add the cache)
>
> The easiest implementation for me would be to be able, from the
> external script, to invalidate the web2py cache whenever I detect changes
> relevant to the active sessions (not the whole cache, just the part
> related
> to my jsonrpc controller)
>
> Is this possible? Maybe by calling a web2py controller which is in
> charge of invalidating the cache?
>
> What about the session? I assume the web2py cache is session-related.
> My script should call the web2py controller using the correct session,
> but
> I do not know how to handle this.
>
> Thanks,
> Daniel
>
--
[web2py] Re: Invalidated RAM cache from outside web2py to handle refreshing of asynchronous data
Thanks Niphlod. With that I can proceed to implementation.
I am also considering another option: use memcached. This is not integrated
with web2py (as far as I know), but would maybe simplify my implementation.
Do you think the performance impact of using memcached (which is accessed
via tcp) compared to using web2py's cache.ram (which lives in the web2py
memory map) is very big?
On Friday, January 11, 2013 3:06:40 PM UTC+1, Niphlod wrote:
>
> 1. if you use cache.ram you can't access the ram that your web2py process
> is using from outside that process.
>In that case you can code a controller that does the trick and call
> it, e.g., with curl from the commandline.
> 2. same thing, curl from the commandline, or use urllib2, or something
> that basically does a request to your webserver.
>
> Il giorno venerdì 11 gennaio 2013 14:53:27 UTC+1, Daniel Gonzalez ha
> scritto:
>>
>> Thanks, that makes sense. In that context, I have two questions:
>>
>>1. How do I call "cache.ram.clear" from outside web2py. What web2py
>>modules should I include?
>>2. How can I call a controller disguised as a user? I think this
>>would be the most robust way, if it is possible ...
>>
>>
>> On Friday, January 11, 2013 2:42:37 PM UTC+1, Niphlod wrote:
>>>
>>> Cache is always "related" to a key. In your function, that key is
>>> request.env.path_info
>>> Let's assume your key is "*the_key*"
>>> If you need to "invalidate" the cache, you should use cache.ram.clear("*
>>> the_key*") or cache.ram("*the_key*", None). From here one, there are 2
>>> paths ahead of you:
>>> - you have an empty cache and it will be filled at the 1st request
>>> coming from an user after you called clear()
>>> - you want to fill that key after clear() ASAP
>>>
>>> In the 1st case, after calling clear() you don't have to do nothing.
>>>
>>> In the 2nd case, either you call the controller "disguised as a user" or
>>> you process the data and store it using cache.ram("*the_key*", thevalue)
>>>
>>> Il giorno venerdì 11 gennaio 2013 13:07:08 UTC+1, Daniel Gonzalez ha
>>> scritto:
Hi,
I am serving some data which is expensive to compute with
a @service.jsonrpc controller. I want to cache the result in RAM, using
something like:
@cache(request.env.path_info,time_expire=1000,cache_model=cache.ram)
My data will not change very often, so I want to have a big ttl
(actually, I would like a ttl "forever", is this possible?)
But my main problem is this: in my architecture, web2py does not know
when new data is available (the data is in a third-party database,
completely out of control of web2py). Changes to the data happen
asynchronously, according to business processes.
I have three pieces:
1. web2py serving the data to the clients, via jsonrpc
2. a small script which knows (long-polling) when changes in the
third-party database occur
3. a library to process the information in the third-party
database. Calling this must be avoided at all costs, *except* when we
have
actual changes (which is why I want to add the cache)
The easiest implementation for me would be to be able, from the
external script, to invalidate the web2py cache whenever I detect changes
relevant to the active sessions (not the whole cache, just the part
related
to my jsonrpc controller)
Is this possible? Maybe by calling a web2py controller which is in
charge of invalidating the cache?
What about the session? I assume the web2py cache is session-related.
My script should call the web2py controller using the correct session, but
I do not know how to handle this.
Thanks,
Daniel
>>>
--
[web2py] Re: Invalidated RAM cache from outside web2py to handle refreshing of asynchronous data
1. if you use cache.ram you can't access the ram that your web2py process
is using from outside that process.
In that case you can code a controller that does the trick and call
it, e.g., with curl from the commandline.
2. same thing, curl from the commandline, or use urllib2, or something that
basically does a request to your webserver.
Il giorno venerdì 11 gennaio 2013 14:53:27 UTC+1, Daniel Gonzalez ha
scritto:
>
> Thanks, that makes sense. In that context, I have two questions:
>
>1. How do I call "cache.ram.clear" from outside web2py. What web2py
>modules should I include?
>2. How can I call a controller disguised as a user? I think this would
>be the most robust way, if it is possible ...
>
>
> On Friday, January 11, 2013 2:42:37 PM UTC+1, Niphlod wrote:
>>
>> Cache is always "related" to a key. In your function, that key is request
>> .env.path_info
>> Let's assume your key is "*the_key*"
>> If you need to "invalidate" the cache, you should use cache.ram.clear("*
>> the_key*") or cache.ram("*the_key*", None). From here one, there are 2
>> paths ahead of you:
>> - you have an empty cache and it will be filled at the 1st request coming
>> from an user after you called clear()
>> - you want to fill that key after clear() ASAP
>>
>> In the 1st case, after calling clear() you don't have to do nothing.
>>
>> In the 2nd case, either you call the controller "disguised as a user" or
>> you process the data and store it using cache.ram("*the_key*", thevalue)
>>
>> Il giorno venerdì 11 gennaio 2013 13:07:08 UTC+1, Daniel Gonzalez ha
>> scritto:
>>>
>>> Hi,
>>>
>>> I am serving some data which is expensive to compute with
>>> a @service.jsonrpc controller. I want to cache the result in RAM, using
>>> something like:
>>>
>>> @cache(request.env.path_info,time_expire=1000,cache_model=cache.ram)
>>>
>>> My data will not change very often, so I want to have a big ttl
>>> (actually, I would like a ttl "forever", is this possible?)
>>>
>>> But my main problem is this: in my architecture, web2py does not know
>>> when new data is available (the data is in a third-party database,
>>> completely out of control of web2py). Changes to the data happen
>>> asynchronously, according to business processes.
>>>
>>> I have three pieces:
>>>
>>>1. web2py serving the data to the clients, via jsonrpc
>>>2. a small script which knows (long-polling) when changes in the
>>>third-party database occur
>>>3. a library to process the information in the third-party database.
>>>Calling this must be avoided at all costs, *except* when we have actual
>>>changes (which is why I want to add the cache)
>>>
>>> The easiest implementation for me would be to be able, from the external
>>> script, to invalidate the web2py cache whenever I detect changes relevant
>>> to the active sessions (not the whole cache, just the part related to my
>>> jsonrpc controller)
>>>
>>> Is this possible? Maybe by calling a web2py controller which is in
>>> charge of invalidating the cache?
>>>
>>> What about the session? I assume the web2py cache is session-related. My
>>> script should call the web2py controller using the correct session, but I
>>> do not know how to handle this.
>>>
>>> Thanks,
>>> Daniel
>>>
>>
--
[web2py] Re: Invalidated RAM cache from outside web2py to handle refreshing of asynchronous data
Thanks, that makes sense. In that context, I have two questions:
1. How do I call "cache.ram.clear" from outside web2py. What web2py
modules should I include?
2. How can I call a controller disguised as a user? I think this would
be the most robust way, if it is possible ...
On Friday, January 11, 2013 2:42:37 PM UTC+1, Niphlod wrote:
>
> Cache is always "related" to a key. In your function, that key is request.
> env.path_info
> Let's assume your key is "*the_key*"
> If you need to "invalidate" the cache, you should use cache.ram.clear("*
> the_key*") or cache.ram("*the_key*", None). From here one, there are 2
> paths ahead of you:
> - you have an empty cache and it will be filled at the 1st request coming
> from an user after you called clear()
> - you want to fill that key after clear() ASAP
>
> In the 1st case, after calling clear() you don't have to do nothing.
>
> In the 2nd case, either you call the controller "disguised as a user" or
> you process the data and store it using cache.ram("*the_key*", thevalue)
>
> Il giorno venerdì 11 gennaio 2013 13:07:08 UTC+1, Daniel Gonzalez ha
> scritto:
>>
>> Hi,
>>
>> I am serving some data which is expensive to compute with
>> a @service.jsonrpc controller. I want to cache the result in RAM, using
>> something like:
>>
>> @cache(request.env.path_info,time_expire=1000,cache_model=cache.ram)
>>
>> My data will not change very often, so I want to have a big ttl
>> (actually, I would like a ttl "forever", is this possible?)
>>
>> But my main problem is this: in my architecture, web2py does not know
>> when new data is available (the data is in a third-party database,
>> completely out of control of web2py). Changes to the data happen
>> asynchronously, according to business processes.
>>
>> I have three pieces:
>>
>>1. web2py serving the data to the clients, via jsonrpc
>>2. a small script which knows (long-polling) when changes in the
>>third-party database occur
>>3. a library to process the information in the third-party database.
>>Calling this must be avoided at all costs, *except* when we have actual
>>changes (which is why I want to add the cache)
>>
>> The easiest implementation for me would be to be able, from the external
>> script, to invalidate the web2py cache whenever I detect changes relevant
>> to the active sessions (not the whole cache, just the part related to my
>> jsonrpc controller)
>>
>> Is this possible? Maybe by calling a web2py controller which is in charge
>> of invalidating the cache?
>>
>> What about the session? I assume the web2py cache is session-related. My
>> script should call the web2py controller using the correct session, but I
>> do not know how to handle this.
>>
>> Thanks,
>> Daniel
>>
>
--
[web2py] Re: Invalidated RAM cache from outside web2py to handle refreshing of asynchronous data
Cache is always "related" to a key. In your function, that key is request.
env.path_info
Let's assume your key is "*the_key*"
If you need to "invalidate" the cache, you should use cache.ram.clear("*
the_key*") or cache.ram("*the_key*", None). From here one, there are 2
paths ahead of you:
- you have an empty cache and it will be filled at the 1st request coming
from an user after you called clear()
- you want to fill that key after clear() ASAP
In the 1st case, after calling clear() you don't have to do nothing.
In the 2nd case, either you call the controller "disguised as a user" or
you process the data and store it using cache.ram("*the_key*", thevalue)
Il giorno venerdì 11 gennaio 2013 13:07:08 UTC+1, Daniel Gonzalez ha
scritto:
>
> Hi,
>
> I am serving some data which is expensive to compute with
> a @service.jsonrpc controller. I want to cache the result in RAM, using
> something like:
>
> @cache(request.env.path_info,time_expire=1000,cache_model=cache.ram)
>
> My data will not change very often, so I want to have a big ttl (actually,
> I would like a ttl "forever", is this possible?)
>
> But my main problem is this: in my architecture, web2py does not know when
> new data is available (the data is in a third-party database, completely
> out of control of web2py). Changes to the data happen asynchronously,
> according to business processes.
>
> I have three pieces:
>
>1. web2py serving the data to the clients, via jsonrpc
>2. a small script which knows (long-polling) when changes in the
>third-party database occur
>3. a library to process the information in the third-party database.
>Calling this must be avoided at all costs, *except* when we have actual
>changes (which is why I want to add the cache)
>
> The easiest implementation for me would be to be able, from the external
> script, to invalidate the web2py cache whenever I detect changes relevant
> to the active sessions (not the whole cache, just the part related to my
> jsonrpc controller)
>
> Is this possible? Maybe by calling a web2py controller which is in charge
> of invalidating the cache?
>
> What about the session? I assume the web2py cache is session-related. My
> script should call the web2py controller using the correct session, but I
> do not know how to handle this.
>
> Thanks,
> Daniel
>
--
